2025-08-18 01:43:12,032 - DEBUG - [10.0.0.49]: Running cmd : hostname 2025-08-18 01:43:12,278 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-237-1 2025-08-18 01:43:12,278 - DEBUG - [10.0.0.49]: Running cmd : hostname -f 2025-08-18 01:43:12,391 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-237-1.cluster.local 2025-08-18 01:43:12,391 - DEBUG - [10.0.0.49]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-18 01:43:12,538 - DEBUG - Output : NAMES contrail_test_Dr4YgWlnk k8s_autoscaler_dns-autoscaler-595558c478-ws96z_kube-system_649d4006-101d-4b97-a990-4399cb30b450_0 k8s_coredns_coredns-77f7cc69db-j98tv_kube-system_c96e12ba-f997-4b0c-a321-db20c88dc8d9_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-9qhdf_kube-system_6795aed9-ac88-4650-ae07-86dbc1b9cb23_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-9qhdf_kube-system_6795aed9-ac88-4650-ae07-86dbc1b9cb23_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-qmg8l_kube-system_1eea9d66-55bd-4243-9128-de79a4097baf_0 k8s_opensdn-controller-control-named_opensdn-controller-control-qmg8l_kube-system_1eea9d66-55bd-4243-9128-de79a4097baf_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-5tm65_kube-system_13e586c2-39d2-49f7-99a1-622a1cf5fd16_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-9qhdf_kube-system_6795aed9-ac88-4650-ae07-86dbc1b9cb23_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-2ltw8_kube-system_3c0dd27f-d46a-4c3c-bf4a-b7ba1a2a0eeb_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-2t5tv_kube-system_777de6d1-989a-4ad8-880f-b1a45cdd3ef2_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-2ltw8_kube-system_3c0dd27f-d46a-4c3c-bf4a-b7ba1a2a0eeb_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-qmg8l_kube-system_1eea9d66-55bd-4243-9128-de79a4097baf_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-qkc96_kube-system_858a0866-017c-422d-808c-7f6ffe8b5536_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-qkc96_kube-system_858a0866-017c-422d-808c-7f6ffe8b5536_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-5tm65_kube-system_13e586c2-39d2-49f7-99a1-622a1cf5fd16_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-54gw2_kube-system_c6e6c3ec-a6be-4dcb-b866-6bfbf0487669_0 k8s_opensdn-analytics-collector_opensdn-analytics-54gw2_kube-system_c6e6c3ec-a6be-4dcb-b866-6bfbf0487669_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-9qhdf_kube-system_6795aed9-ac88-4650-ae07-86dbc1b9cb23_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-2t5tv_kube-system_777de6d1-989a-4ad8-880f-b1a45cdd3ef2_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-2ltw8_kube-system_3c0dd27f-d46a-4c3c-bf4a-b7ba1a2a0eeb_0 k8s_opensdn-controller-control_opensdn-controller-control-qmg8l_kube-system_1eea9d66-55bd-4243-9128-de79a4097baf_0 k8s_kafka_opensdn-analytics-alarm-qkc96_kube-system_858a0866-017c-422d-808c-7f6ffe8b5536_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-5tm65_kube-system_13e586c2-39d2-49f7-99a1-622a1cf5fd16_0 k8s_opensdn-analytics-api_opensdn-analytics-54gw2_kube-system_c6e6c3ec-a6be-4dcb-b866-6bfbf0487669_0 k8s_opensdn-controller-config-api_opensdn-controller-config-9qhdf_kube-system_6795aed9-ac88-4650-ae07-86dbc1b9cb23_0 k8s_config-zookeeper_config-zookeeper-zntgf_kube-system_f4bf34d5-6339-41f8-a83c-f0aec46146c9_0 k8s_rabbitmq_rabbitmq-2qq5r_kube-system_08ec67e4-00a1-45e4-98b7-dc9bc2f11bc8_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-mcfpj_kube-system_89bdadf8-c800-41c4-9849-6bc85fa0cf01_0 k8s_opensdn-configdb_opensdn-configdb-mcfpj_kube-system_89bdadf8-c800-41c4-9849-6bc85fa0cf01_0 k8s_redis_redis-tcfrx_kube-system_d96a018a-640c-4aee-9bd9-cef129d4a0b4_0 k8s_opensdn-agent-nodemgr_opensdn-agent-vqlvx_kube-system_49bf007f-6988-4310-894c-12fa47876363_0 k8s_opensdn-vrouter-agent_opensdn-agent-vqlvx_kube-system_49bf007f-6988-4310-894c-12fa47876363_0 k8s_opensdn-kube-manager_opensdn-kube-manager-z4rb7_kube-system_3d81a698-4c98-4f8f-b374-cc36e33d5ef2_0 k8s_kube-proxy_kube-proxy-dzdln_kube-system_cb5c8cd6-a604-476f-899f-ca3be67eeb72_0 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-237-1_kube-system_7490ef7aaa953e9ee786848f001d2810_1 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-237-1_kube-system_9557af228936146ef18bbfc07c71788d_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-237-1_kube-system_80ffa11c3bbf22dc7c36a1c2a713a9ed_0 2025-08-18 01:43:12,539 - DEBUG - [10.0.0.49]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-18 01:43:12,642 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.1/32 10.96.0.3/32 10.0.0.49/24 2025-08-18 01:43:12,643 - DEBUG - [10.0.0.49]: Running cmd : getent hosts 10.0.0.49 | head -n 1 | awk '{print $2}' 2025-08-18 01:43:12,755 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-237-1.cluster.local 2025-08-18 01:43:12,755 - DEBUG - [10.0.0.49]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-08-18 01:43:12,864 - DEBUG - Output : 10.0.0.49/24 2025-08-18 01:43:13,227 - INFO - Using existing project ['default-domain', 'k8s-default'](967038f2-0c51-4786-8770-eeccfadbdcec) 2025-08-18 01:43:13,411 - INFO - Using existing project ['default-domain', 'k8s-default'](967038f2-0c51-4786-8770-eeccfadbdcec) 2025-08-18 01:43:13,596 - INFO - ================================================================================ 2025-08-18 01:43:13,596 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-08-18 01:43:13,597 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-08-18 01:43:13,907 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-18 01:43:13,914 - DEBUG - Nothing to compare xmpp stats {'10.0.0.49': {'10.0.0.49': '2'}} with 2025-08-18 01:43:13,914 - INFO - Initial checks done. Running the testcase now 2025-08-18 01:43:13,914 - INFO - 2025-08-18 01:43:13,919 - INFO - Namespace default already exists 2025-08-18 01:43:13,919 - DEBUG - [10.0.0.49]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-z4rb7_kube-system_3d81a698-4c98-4f8f-b374-cc36e33d5ef2_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-08-18 01:43:14,103 - DEBUG - Output : cluster_project={} 2025-08-18 01:43:14,170 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-08-18 01:43:14,170 - DEBUG - Project uuid in Contrail is 967038f2-0c51-4786-8770-eeccfadbdcec 2025-08-18 01:43:14,171 - DEBUG - [10.0.0.49]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-z4rb7_kube-system_3d81a698-4c98-4f8f-b374-cc36e33d5ef2_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-08-18 01:43:14,307 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2025-08-18 01:43:14,308 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_NamespaceDatabaseList?namespace_uuid=7a2c4f80-0be8-485f-a075-6457a56e45ae 2025-08-18 01:43:14,313 - INFO - Namespace default with uuid 7a2c4f80-0be8-485f-a075-6457a56e45ae found in kube manager 2025-08-18 01:43:14,313 - INFO - Namespace default verification passed 2025-08-18 01:43:14,316 - DEBUG - Service ctest-nginx-svc-80250006 not present 2025-08-18 01:43:14,317 - INFO - Creating service ctest-nginx-svc-80250006 2025-08-18 01:43:14,330 - INFO - Created Service ctest-nginx-svc-80250006 2025-08-18 01:43:14,337 - DEBUG - Pod ctest-nginx-pod-41377514 not present 2025-08-18 01:43:14,337 - INFO - Creating Pod ctest-nginx-pod-41377514 2025-08-18 01:43:14,356 - DEBUG - Pod : ctest-nginx-pod-41377514 UUID is 83c189c1-d245-4e42-a5a0-5d9345f73698 2025-08-18 01:43:14,359 - DEBUG - Pod ctest-nginx-pod-91871617 not present 2025-08-18 01:43:14,359 - INFO - Creating Pod ctest-nginx-pod-91871617 2025-08-18 01:43:14,373 - DEBUG - Pod : ctest-nginx-pod-91871617 UUID is c0da5794-50b1-4044-97e9-21a6ea5a524e 2025-08-18 01:43:15,035 - INFO - Created VN __public__, UUID :416ff8c1-1cda-43fa-b6fe-90c5b0a2abc6 2025-08-18 01:43:15,370 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-08-18 01:43:15,376 - DEBUG - Requesting: http://10.0.0.49:8082/domain/9d3bb778-f98d-4bad-8479-c46fdd4d04d3 2025-08-18 01:43:15,419 - DEBUG - Requesting: http://10.0.0.49:8082/project/967038f2-0c51-4786-8770-eeccfadbdcec 2025-08-18 01:43:15,605 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/416ff8c1-1cda-43fa-b6fe-90c5b0a2abc6 2025-08-18 01:43:15,621 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/416ff8c1-1cda-43fa-b6fe-90c5b0a2abc6 2025-08-18 01:43:15,636 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/0208cdd2-7321-4def-977d-f03996b7dddc 2025-08-18 01:43:15,648 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/0208cdd2-7321-4def-977d-f03996b7dddc 2025-08-18 01:43:15,691 - DEBUG - Requesting: http://10.0.0.49:8082/route-target/d980bf87-b76b-4a24-8431-0f02e25b127f 2025-08-18 01:43:15,708 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-18 01:43:15,708 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/416ff8c1-1cda-43fa-b6fe-90c5b0a2abc6 2025-08-18 01:43:15,735 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/0208cdd2-7321-4def-977d-f03996b7dddc 2025-08-18 01:43:15,800 - INFO - Verified VN network id 12 for VN 416ff8c1-1cda-43fa-b6fe-90c5b0a2abc6 2025-08-18 01:43:15,800 - INFO - Verifications in API Server for VN __public__ passed 2025-08-18 01:43:15,800 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/416ff8c1-1cda-43fa-b6fe-90c5b0a2abc6 2025-08-18 01:43:15,816 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/0208cdd2-7321-4def-977d-f03996b7dddc 2025-08-18 01:43:15,831 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/0208cdd2-7321-4def-977d-f03996b7dddc 2025-08-18 01:43:15,850 - DEBUG - Requesting: http://10.0.0.49:8082/route-target/d980bf87-b76b-4a24-8431-0f02e25b127f 2025-08-18 01:43:15,861 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-08-18 01:43:15,871 - DEBUG - Control-node 10.0.0.49 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4715260843176641530', 'uuid-lslong': '13186135937733798854'}, 'enable': 'true', 'created': '2025-08-18T01:43:14', 'last-modified': '2025-08-18T01:43:15', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.625629'} 2025-08-18 01:43:15,871 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-18 01:43:15,878 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-18 01:43:15,878 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2025-08-18 01:43:15,879 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2025-08-18 01:43:15,879 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-08-18 01:43:15,887 - DEBUG - Requesting: http://10.0.0.49:8082/domain/9d3bb778-f98d-4bad-8479-c46fdd4d04d3 2025-08-18 01:43:15,896 - DEBUG - Requesting: http://10.0.0.49:8082/project/967038f2-0c51-4786-8770-eeccfadbdcec 2025-08-18 01:43:15,967 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/416ff8c1-1cda-43fa-b6fe-90c5b0a2abc6 2025-08-18 01:43:15,989 - DEBUG - =>VN __public__ has no policy to be verified 2025-08-18 01:43:15,989 - DEBUG - Verifying the vn in opserver 2025-08-18 01:43:15,989 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.49 2025-08-18 01:43:15,990 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-networks 2025-08-18 01:43:15,996 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.49:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2025-08-18 01:43:15,996 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2025-08-18 01:43:15,996 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-18 01:43:16,005 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-18 01:43:16,014 - DEBUG - Do not have enough data to verify VN in agent 2025-08-18 01:43:16,014 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-18 01:43:16,021 - DEBUG - VRF ids for VN __public__: {} 2025-08-18 01:43:16,193 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2025-08-18 01:43:16,402 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-08-18 01:43:16,408 - DEBUG - Requesting: http://10.0.0.49:8082/domain/9d3bb778-f98d-4bad-8479-c46fdd4d04d3 2025-08-18 01:43:16,415 - DEBUG - Requesting: http://10.0.0.49:8082/project/967038f2-0c51-4786-8770-eeccfadbdcec 2025-08-18 01:43:16,455 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/416ff8c1-1cda-43fa-b6fe-90c5b0a2abc6 2025-08-18 01:43:16,465 - DEBUG - Requesting: http://10.0.0.49:8082/floating-ip-pool/b36bc68a-f64f-469e-9372-646ba22053b2 2025-08-18 01:43:16,478 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-08-18 01:43:16,485 - DEBUG - Requesting: http://10.0.0.49:8082/domain/9d3bb778-f98d-4bad-8479-c46fdd4d04d3 2025-08-18 01:43:16,492 - DEBUG - Requesting: http://10.0.0.49:8082/project/967038f2-0c51-4786-8770-eeccfadbdcec 2025-08-18 01:43:16,528 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/416ff8c1-1cda-43fa-b6fe-90c5b0a2abc6 2025-08-18 01:43:16,540 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2025-08-18 01:43:16,540 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2025-08-18 01:43:16,547 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-08-18 01:43:16,553 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2025-08-18 01:43:16,553 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2025-08-18 01:43:16,556 - DEBUG - Ingress ctest-nginx-ingress-60486650 not present 2025-08-18 01:43:16,557 - INFO - Creating Ingress ctest-nginx-ingress-60486650 2025-08-18 01:43:16,563 - INFO - Created Ingress ctest-nginx-ingress-60486650 2025-08-18 01:43:16,566 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-60486650 2025-08-18 01:43:19,570 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-60486650 2025-08-18 01:43:22,576 - DEBUG - For Ingress ctest-nginx-ingress-60486650, Cluster IP: 10.47.255.248, External IPs ['128.81.9.124'] 2025-08-18 01:43:22,577 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-60486650 2025-08-18 01:43:22,577 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=d2171b6e-890c-42b5-8816-8de55cafaf8a 2025-08-18 01:43:22,585 - INFO - Ingress ctest-nginx-ingress-60486650 with uuid d2171b6e-890c-42b5-8816-8de55cafaf8a found in kube manager 2025-08-18 01:43:22,585 - INFO - Ingress ctest-nginx-ingress-60486650 verification passed 2025-08-18 01:43:22,589 - DEBUG - Pod ctest-busybox-pod-73961067 not present 2025-08-18 01:43:22,589 - INFO - Creating Pod ctest-busybox-pod-73961067 2025-08-18 01:43:22,609 - DEBUG - Pod : ctest-busybox-pod-73961067 UUID is 01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f 2025-08-18 01:43:22,618 - DEBUG - Pod ctest-nginx-pod-41377514 not in running state.Currently in Pending 2025-08-18 01:43:27,632 - DEBUG - Pod ctest-nginx-pod-41377514 not in running state.Currently in Pending 2025-08-18 01:43:32,640 - INFO - Pod ctest-nginx-pod-41377514 is in running state.Got IP 10.47.255.250 2025-08-18 01:43:32,698 - DEBUG - Pod ctest-nginx-pod-41377514 has vmi b66a9fde-7bd4-11f0-b104-02425efb73c6 2025-08-18 01:43:32,698 - INFO - Verified pod ctest-nginx-pod-41377514 in contrail-api 2025-08-18 01:43:32,698 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-08-18 01:43:32,711 - DEBUG - VMI b66a9fde-7bd4-11f0-b104-02425efb73c6 is active in agent 10.0.0.49 2025-08-18 01:43:32,711 - INFO - Verified Pod ctest-nginx-pod-41377514 in agent 10.0.0.49 2025-08-18 01:43:32,711 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_VirtualMachineDatabaseList?x=83c189c1-d245-4e42-a5a0-5d9345f73698 2025-08-18 01:43:32,716 - INFO - Pod ctest-nginx-pod-41377514 with uuid 83c189c1-d245-4e42-a5a0-5d9345f73698 found in kube manager 2025-08-18 01:43:32,716 - INFO - Pod ctest-nginx-pod-41377514 verification passed 2025-08-18 01:43:32,800 - DEBUG - [Pod ctest-nginx-pod-41377514] Cmd: echo ctest-nginx-pod-41377514 > /usr/share/nginx/html/index.html, Output: 2025-08-18 01:43:32,806 - INFO - Pod ctest-nginx-pod-91871617 is in running state.Got IP 10.47.255.249 2025-08-18 01:43:32,860 - DEBUG - Pod ctest-nginx-pod-91871617 has vmi b71993ea-7bd4-11f0-b104-02425efb73c6 2025-08-18 01:43:32,860 - INFO - Verified pod ctest-nginx-pod-91871617 in contrail-api 2025-08-18 01:43:32,860 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-08-18 01:43:32,870 - DEBUG - VMI b71993ea-7bd4-11f0-b104-02425efb73c6 is active in agent 10.0.0.49 2025-08-18 01:43:32,870 - INFO - Verified Pod ctest-nginx-pod-91871617 in agent 10.0.0.49 2025-08-18 01:43:32,870 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_VirtualMachineDatabaseList?x=c0da5794-50b1-4044-97e9-21a6ea5a524e 2025-08-18 01:43:32,874 - INFO - Pod ctest-nginx-pod-91871617 with uuid c0da5794-50b1-4044-97e9-21a6ea5a524e found in kube manager 2025-08-18 01:43:32,874 - INFO - Pod ctest-nginx-pod-91871617 verification passed 2025-08-18 01:43:32,942 - DEBUG - [Pod ctest-nginx-pod-91871617] Cmd: echo ctest-nginx-pod-91871617 > /usr/share/nginx/html/index.html, Output: 2025-08-18 01:43:32,948 - DEBUG - Pod ctest-busybox-pod-73961067 not in running state.Currently in Pending 2025-08-18 01:43:37,959 - INFO - Pod ctest-busybox-pod-73961067 is in running state.Got IP 10.47.255.247 2025-08-18 01:43:38,014 - DEBUG - Pod ctest-busybox-pod-73961067 has vmi ba8c0fd0-7bd4-11f0-b104-02425efb73c6 2025-08-18 01:43:38,014 - INFO - Verified pod ctest-busybox-pod-73961067 in contrail-api 2025-08-18 01:43:38,014 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-08-18 01:43:38,028 - DEBUG - VMI ba8c0fd0-7bd4-11f0-b104-02425efb73c6 is active in agent 10.0.0.49 2025-08-18 01:43:38,028 - INFO - Verified Pod ctest-busybox-pod-73961067 in agent 10.0.0.49 2025-08-18 01:43:38,028 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_VirtualMachineDatabaseList?x=01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f 2025-08-18 01:43:38,037 - INFO - Pod ctest-busybox-pod-73961067 with uuid 01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f found in kube manager 2025-08-18 01:43:38,037 - INFO - Pod ctest-busybox-pod-73961067 verification passed 2025-08-18 01:43:38,116 - DEBUG - [Pod ctest-busybox-pod-73961067] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-41377514 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-08-18 01:43:38,116 - DEBUG - [Pod Pod ctest-busybox-pod-73961067] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-08-18 01:43:38,116 - DEBUG - [Pod Pod ctest-busybox-pod-73961067] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-41377514 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-08-18 01:43:38,202 - DEBUG - [Pod ctest-busybox-pod-73961067] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| ctest-nginx-pod-91871617 25 0:00:00 ETA written to stdout 2025-08-18 01:43:38,203 - DEBUG - [Pod Pod ctest-busybox-pod-73961067] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-08-18 01:43:38,203 - DEBUG - [Pod Pod ctest-busybox-pod-73961067] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| ctest-nginx-pod-91871617 25 0:00:00 ETA written to stdout 2025-08-18 01:43:38,203 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-41377514': 1, 'ctest-nginx-pod-91871617': 1} 2025-08-18 01:43:38,203 - WARNING - Nothing to delete parallely 2025-08-18 01:43:38,203 - INFO - Deleting pod default:ctest-busybox-pod-73961067 2025-08-18 01:43:38,217 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f 2025-08-18 01:43:38,234 - WARNING - Pod uuid 01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f is still seen in agent 10.0.0.49 VM list 2025-08-18 01:43:43,235 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f 2025-08-18 01:43:43,241 - WARNING - Pod uuid 01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f is still seen in agent 10.0.0.49 VM list 2025-08-18 01:43:48,242 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f 2025-08-18 01:43:48,248 - WARNING - Pod uuid 01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f is still seen in agent 10.0.0.49 VM list 2025-08-18 01:43:53,249 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f 2025-08-18 01:43:53,258 - WARNING - Pod uuid 01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f is still seen in agent 10.0.0.49 VM list 2025-08-18 01:43:58,260 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f 2025-08-18 01:43:58,266 - WARNING - Pod uuid 01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f is still seen in agent 10.0.0.49 VM list 2025-08-18 01:44:03,267 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f 2025-08-18 01:44:03,273 - WARNING - Pod uuid 01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f is still seen in agent 10.0.0.49 VM list 2025-08-18 01:44:08,274 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f 2025-08-18 01:44:08,279 - WARNING - Pod uuid 01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f is still seen in agent 10.0.0.49 VM list 2025-08-18 01:44:13,280 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f 2025-08-18 01:44:13,286 - DEBUG - Pod 01f0bbfd-8e42-4a4b-9fbc-7f5e9974d13f is not in agent 10.0.0.49 VM list 2025-08-18 01:44:13,286 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ba8c0fd0-7bd4-11f0-b104-02425efb73c6 2025-08-18 01:44:13,291 - DEBUG - VMI ba8c0fd0-7bd4-11f0-b104-02425efb73c6 is removed from agent 10.0.0.49 2025-08-18 01:44:13,291 - INFO - Verified that pod ctest-busybox-pod-73961067 is removed in agent 2025-08-18 01:44:13,291 - INFO - Deleting Ingress : ctest-nginx-ingress-60486650 2025-08-18 01:44:13,297 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=d2171b6e-890c-42b5-8816-8de55cafaf8a 2025-08-18 01:44:13,301 - ERROR - Ingress ctest-nginx-ingress-60486650 with uuid d2171b6e-890c-42b5-8816-8de55cafaf8a still found in kube manager 2025-08-18 01:44:14,302 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=d2171b6e-890c-42b5-8816-8de55cafaf8a 2025-08-18 01:44:14,307 - ERROR - Ingress ctest-nginx-ingress-60486650 with uuid d2171b6e-890c-42b5-8816-8de55cafaf8a still found in kube manager 2025-08-18 01:44:15,307 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=d2171b6e-890c-42b5-8816-8de55cafaf8a 2025-08-18 01:44:15,316 - INFO - Ingress ctest-nginx-ingress-60486650 with uuid d2171b6e-890c-42b5-8816-8de55cafaf8a deleted successfully from kube manager 2025-08-18 01:44:15,316 - INFO - Verifications on Ingress %s cleanup passed 2025-08-18 01:44:15,316 - INFO - Deleting the FIP pool __fip_pool_public__ 2025-08-18 01:44:15,428 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-08-18 01:44:15,439 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2025-08-18 01:44:15,439 - INFO - Deleting VN __public__ 2025-08-18 01:44:15,545 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/0208cdd2-7321-4def-977d-f03996b7dddc 2025-08-18 01:44:15,549 - DEBUG - Response Code: 404 2025-08-18 01:44:15,549 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-08-18 01:44:15,554 - DEBUG - Requesting: http://10.0.0.49:8082/domain/9d3bb778-f98d-4bad-8479-c46fdd4d04d3 2025-08-18 01:44:15,561 - DEBUG - Requesting: http://10.0.0.49:8082/project/967038f2-0c51-4786-8770-eeccfadbdcec 2025-08-18 01:44:15,614 - INFO - Validated that VN __public__ is not found in API Server 2025-08-18 01:44:15,614 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2025-08-18 01:44:15,620 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-18 01:44:15,626 - DEBUG - VN __public__ is not present in Agent 10.0.0.49 2025-08-18 01:44:15,626 - INFO - Validated that VN __public__ is not in any agent 2025-08-18 01:44:15,626 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-18 01:44:15,632 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-18 01:44:15,638 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-08-18 01:44:15,643 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2025-08-18 01:44:15,643 - INFO - Deleting pod default:ctest-nginx-pod-91871617 2025-08-18 01:44:15,654 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=c0da5794-50b1-4044-97e9-21a6ea5a524e 2025-08-18 01:44:15,664 - WARNING - Pod uuid c0da5794-50b1-4044-97e9-21a6ea5a524e is still seen in agent 10.0.0.49 VM list 2025-08-18 01:44:20,665 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=c0da5794-50b1-4044-97e9-21a6ea5a524e 2025-08-18 01:44:20,672 - DEBUG - Pod c0da5794-50b1-4044-97e9-21a6ea5a524e is not in agent 10.0.0.49 VM list 2025-08-18 01:44:20,672 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b71993ea-7bd4-11f0-b104-02425efb73c6 2025-08-18 01:44:20,677 - DEBUG - VMI b71993ea-7bd4-11f0-b104-02425efb73c6 is removed from agent 10.0.0.49 2025-08-18 01:44:20,677 - INFO - Verified that pod ctest-nginx-pod-91871617 is removed in agent 2025-08-18 01:44:20,677 - INFO - Deleting pod default:ctest-nginx-pod-41377514 2025-08-18 01:44:20,696 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=83c189c1-d245-4e42-a5a0-5d9345f73698 2025-08-18 01:44:20,708 - WARNING - Pod uuid 83c189c1-d245-4e42-a5a0-5d9345f73698 is still seen in agent 10.0.0.49 VM list 2025-08-18 01:44:25,712 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=83c189c1-d245-4e42-a5a0-5d9345f73698 2025-08-18 01:44:25,720 - DEBUG - Pod 83c189c1-d245-4e42-a5a0-5d9345f73698 is not in agent 10.0.0.49 VM list 2025-08-18 01:44:25,720 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b66a9fde-7bd4-11f0-b104-02425efb73c6 2025-08-18 01:44:25,726 - DEBUG - VMI b66a9fde-7bd4-11f0-b104-02425efb73c6 is removed from agent 10.0.0.49 2025-08-18 01:44:25,726 - INFO - Verified that pod ctest-nginx-pod-41377514 is removed in agent 2025-08-18 01:44:25,726 - INFO - Deleting service : ctest-nginx-svc-80250006 2025-08-18 01:44:26,212 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-18 01:44:26,218 - DEBUG - No XMPP flaps were noticed during the test