2025-08-11 01:40:08,943 - DEBUG - [10.0.0.49]: Running cmd : hostname 2025-08-11 01:40:09,172 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-235-1 2025-08-11 01:40:09,172 - DEBUG - [10.0.0.49]: Running cmd : hostname -f 2025-08-11 01:40:09,306 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-235-1.cluster.local 2025-08-11 01:40:09,306 - DEBUG - [10.0.0.49]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 01:40:09,469 - DEBUG - Output : NAMES contrail_test_I4tkIgsUK k8s_autoscaler_dns-autoscaler-595558c478-d7vh5_kube-system_92cd0fb2-3771-41a6-8d35-e20ae4794bcb_0 k8s_coredns_coredns-77f7cc69db-sclz8_kube-system_f1a6ee48-866a-4fab-a8cf-b230b1ed110e_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-d9zlg_kube-system_2c088a7e-f3df-4d54-9782-30c3e3ca175b_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-d9zlg_kube-system_2c088a7e-f3df-4d54-9782-30c3e3ca175b_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-r7rwt_kube-system_712b79ed-150b-4dc7-99ae-05fec68041b8_0 k8s_opensdn-controller-control-named_opensdn-controller-control-r7rwt_kube-system_712b79ed-150b-4dc7-99ae-05fec68041b8_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-x56mj_kube-system_1e66c7bf-ffa0-46d6-9557-bcdab9d80842_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-d9zlg_kube-system_2c088a7e-f3df-4d54-9782-30c3e3ca175b_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-qbj6c_kube-system_f8d46b61-3cc5-4a5e-821e-475a79d052f3_0 k8s_opensdn-analytics-collector_opensdn-analytics-qbj6c_kube-system_f8d46b61-3cc5-4a5e-821e-475a79d052f3_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-r7rwt_kube-system_712b79ed-150b-4dc7-99ae-05fec68041b8_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-g8mxx_kube-system_eda2eac8-a1fc-44b5-aa08-d856c9efdcf0_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-rpxhz_kube-system_dad844aa-ef0c-4a12-a2f2-34ec88ff1efe_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-g8mxx_kube-system_eda2eac8-a1fc-44b5-aa08-d856c9efdcf0_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-4wzwg_kube-system_215589ce-b1ac-4c18-8704-7dacf9e8dc53_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-4wzwg_kube-system_215589ce-b1ac-4c18-8704-7dacf9e8dc53_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-x56mj_kube-system_1e66c7bf-ffa0-46d6-9557-bcdab9d80842_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-d9zlg_kube-system_2c088a7e-f3df-4d54-9782-30c3e3ca175b_0 k8s_opensdn-analytics-api_opensdn-analytics-qbj6c_kube-system_f8d46b61-3cc5-4a5e-821e-475a79d052f3_0 k8s_opensdn-controller-control_opensdn-controller-control-r7rwt_kube-system_712b79ed-150b-4dc7-99ae-05fec68041b8_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-rpxhz_kube-system_dad844aa-ef0c-4a12-a2f2-34ec88ff1efe_0 k8s_kafka_opensdn-analytics-alarm-g8mxx_kube-system_eda2eac8-a1fc-44b5-aa08-d856c9efdcf0_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-4wzwg_kube-system_215589ce-b1ac-4c18-8704-7dacf9e8dc53_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-x56mj_kube-system_1e66c7bf-ffa0-46d6-9557-bcdab9d80842_0 k8s_opensdn-controller-config-api_opensdn-controller-config-d9zlg_kube-system_2c088a7e-f3df-4d54-9782-30c3e3ca175b_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-drhm6_kube-system_75658358-1244-4cee-9963-1fc942b929f3_0 k8s_opensdn-configdb_opensdn-configdb-drhm6_kube-system_75658358-1244-4cee-9963-1fc942b929f3_0 k8s_rabbitmq_rabbitmq-xc57r_kube-system_a00fd740-ffb2-4b8f-bd04-b2ee087aae02_0 k8s_config-zookeeper_config-zookeeper-ml6b9_kube-system_ac63acba-8095-4a11-9fe0-fe3d03d3c0a0_0 k8s_redis_redis-nhm79_kube-system_31622bcb-af56-4dde-a40c-953975732d7e_0 k8s_opensdn-agent-nodemgr_opensdn-agent-npxp6_kube-system_eb95ace1-13f9-4826-bdf3-8a480d956946_0 k8s_opensdn-vrouter-agent_opensdn-agent-npxp6_kube-system_eb95ace1-13f9-4826-bdf3-8a480d956946_0 k8s_opensdn-kube-manager_opensdn-kube-manager-ll78t_kube-system_ccafba0e-3fb9-440a-9367-a58dc637770a_0 k8s_kube-proxy_kube-proxy-q4nxg_kube-system_946a025a-8d40-4403-9eae-193905f1eb81_0 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-235-1_kube-system_82d14429e710772379b33299f582f094_1 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-235-1_kube-system_a77710353f091ddffc2e5845f2684089_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-235-1_kube-system_ce9232c2f447d96fdc62788c9f254bf3_0 2025-08-11 01:40:09,470 - DEBUG - [10.0.0.49]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-11 01:40:09,583 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.1/32 10.96.0.3/32 10.0.0.49/24 2025-08-11 01:40:09,583 - DEBUG - [10.0.0.49]: Running cmd : getent hosts 10.0.0.49 | head -n 1 | awk '{print $2}' 2025-08-11 01:40:09,695 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-235-1.cluster.local 2025-08-11 01:40:09,695 - DEBUG - [10.0.0.49]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-08-11 01:40:09,806 - DEBUG - Output : 10.0.0.49/24 2025-08-11 01:40:10,175 - INFO - Using existing project ['default-domain', 'k8s-default'](eb9d3b2f-1fed-433d-85c7-dc1998404596) 2025-08-11 01:40:10,330 - INFO - Using existing project ['default-domain', 'k8s-default'](eb9d3b2f-1fed-433d-85c7-dc1998404596) 2025-08-11 01:40:10,511 - INFO - ================================================================================ 2025-08-11 01:40:10,511 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-08-11 01:40:10,512 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-08-11 01:40:10,760 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 01:40:10,767 - DEBUG - Nothing to compare xmpp stats {'10.0.0.49': {'10.0.0.49': '2'}} with 2025-08-11 01:40:10,767 - INFO - Initial checks done. Running the testcase now 2025-08-11 01:40:10,767 - INFO - 2025-08-11 01:40:10,771 - INFO - Namespace default already exists 2025-08-11 01:40:10,771 - DEBUG - [10.0.0.49]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-ll78t_kube-system_ccafba0e-3fb9-440a-9367-a58dc637770a_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-08-11 01:40:10,927 - DEBUG - Output : cluster_project={} 2025-08-11 01:40:10,998 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-08-11 01:40:10,998 - DEBUG - Project uuid in Contrail is eb9d3b2f-1fed-433d-85c7-dc1998404596 2025-08-11 01:40:10,998 - DEBUG - [10.0.0.49]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-ll78t_kube-system_ccafba0e-3fb9-440a-9367-a58dc637770a_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-08-11 01:40:11,136 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2025-08-11 01:40:11,136 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_NamespaceDatabaseList?namespace_uuid=336e2d9a-99f5-4d03-9c9b-3306c2ae2dcd 2025-08-11 01:40:11,141 - INFO - Namespace default with uuid 336e2d9a-99f5-4d03-9c9b-3306c2ae2dcd found in kube manager 2025-08-11 01:40:11,141 - INFO - Namespace default verification passed 2025-08-11 01:40:11,145 - DEBUG - Service ctest-nginx-svc-25695655 not present 2025-08-11 01:40:11,145 - INFO - Creating service ctest-nginx-svc-25695655 2025-08-11 01:40:11,156 - INFO - Created Service ctest-nginx-svc-25695655 2025-08-11 01:40:11,162 - DEBUG - Pod ctest-nginx-pod-46069662 not present 2025-08-11 01:40:11,163 - INFO - Creating Pod ctest-nginx-pod-46069662 2025-08-11 01:40:11,193 - DEBUG - Pod : ctest-nginx-pod-46069662 UUID is 8392971d-a26f-4a1a-9e8a-43212a1158d6 2025-08-11 01:40:11,196 - DEBUG - Pod ctest-nginx-pod-96563660 not present 2025-08-11 01:40:11,196 - INFO - Creating Pod ctest-nginx-pod-96563660 2025-08-11 01:40:11,212 - DEBUG - Pod : ctest-nginx-pod-96563660 UUID is 810e1ce0-ff3b-4649-a3de-66a129e43818 2025-08-11 01:40:11,466 - INFO - Created VN __public__, UUID :ebf405a4-d3f3-4bd9-a2a6-b1df816e0759 2025-08-11 01:40:11,834 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-08-11 01:40:11,841 - DEBUG - Requesting: http://10.0.0.49:8082/domain/85e18e77-68eb-48c5-86da-0dd561703453 2025-08-11 01:40:11,889 - DEBUG - Requesting: http://10.0.0.49:8082/project/eb9d3b2f-1fed-433d-85c7-dc1998404596 2025-08-11 01:40:11,986 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/ebf405a4-d3f3-4bd9-a2a6-b1df816e0759 2025-08-11 01:40:11,996 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/ebf405a4-d3f3-4bd9-a2a6-b1df816e0759 2025-08-11 01:40:12,005 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/159b6f27-5c13-4b97-a14a-09e908aa4c5d 2025-08-11 01:40:12,012 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/159b6f27-5c13-4b97-a14a-09e908aa4c5d 2025-08-11 01:40:12,019 - DEBUG - Requesting: http://10.0.0.49:8082/route-target/ba93580d-255e-4540-940b-982a60323244 2025-08-11 01:40:12,026 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-11 01:40:12,026 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/ebf405a4-d3f3-4bd9-a2a6-b1df816e0759 2025-08-11 01:40:12,036 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/159b6f27-5c13-4b97-a14a-09e908aa4c5d 2025-08-11 01:40:12,053 - INFO - Verified VN network id 12 for VN ebf405a4-d3f3-4bd9-a2a6-b1df816e0759 2025-08-11 01:40:12,053 - INFO - Verifications in API Server for VN __public__ passed 2025-08-11 01:40:12,053 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/ebf405a4-d3f3-4bd9-a2a6-b1df816e0759 2025-08-11 01:40:12,062 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/159b6f27-5c13-4b97-a14a-09e908aa4c5d 2025-08-11 01:40:12,069 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/159b6f27-5c13-4b97-a14a-09e908aa4c5d 2025-08-11 01:40:12,076 - DEBUG - Requesting: http://10.0.0.49:8082/route-target/ba93580d-255e-4540-940b-982a60323244 2025-08-11 01:40:12,082 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-08-11 01:40:12,089 - DEBUG - Control-node 10.0.0.49 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17002220698719177689', 'uuid-lslong': '11720250653785589593'}, 'enable': 'true', 'created': '2025-08-11T01:40:11', 'last-modified': '2025-08-11T01:40:11', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.374719'} 2025-08-11 01:40:12,089 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-11 01:40:12,095 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-11 01:40:12,095 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2025-08-11 01:40:12,096 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2025-08-11 01:40:12,096 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-08-11 01:40:12,100 - DEBUG - Requesting: http://10.0.0.49:8082/domain/85e18e77-68eb-48c5-86da-0dd561703453 2025-08-11 01:40:12,107 - DEBUG - Requesting: http://10.0.0.49:8082/project/eb9d3b2f-1fed-433d-85c7-dc1998404596 2025-08-11 01:40:12,144 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/ebf405a4-d3f3-4bd9-a2a6-b1df816e0759 2025-08-11 01:40:12,153 - DEBUG - =>VN __public__ has no policy to be verified 2025-08-11 01:40:12,153 - DEBUG - Verifying the vn in opserver 2025-08-11 01:40:12,153 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.49 2025-08-11 01:40:12,154 - DEBUG - Requesting: http://10.0.0.49:8081/analytics/uves/virtual-networks 2025-08-11 01:40:12,159 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.49:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2025-08-11 01:40:12,159 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2025-08-11 01:40:12,159 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-11 01:40:12,166 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-11 01:40:12,171 - DEBUG - Do not have enough data to verify VN in agent 2025-08-11 01:40:12,171 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-11 01:40:12,175 - DEBUG - VRF ids for VN __public__: {} 2025-08-11 01:40:12,352 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2025-08-11 01:40:12,709 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-08-11 01:40:12,724 - DEBUG - Requesting: http://10.0.0.49:8082/domain/85e18e77-68eb-48c5-86da-0dd561703453 2025-08-11 01:40:12,731 - DEBUG - Requesting: http://10.0.0.49:8082/project/eb9d3b2f-1fed-433d-85c7-dc1998404596 2025-08-11 01:40:12,769 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/ebf405a4-d3f3-4bd9-a2a6-b1df816e0759 2025-08-11 01:40:12,783 - DEBUG - Requesting: http://10.0.0.49:8082/floating-ip-pool/1bd7eb20-e7f5-47e8-8a87-4bc685f0265b 2025-08-11 01:40:12,791 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-08-11 01:40:12,799 - DEBUG - Requesting: http://10.0.0.49:8082/domain/85e18e77-68eb-48c5-86da-0dd561703453 2025-08-11 01:40:12,807 - DEBUG - Requesting: http://10.0.0.49:8082/project/eb9d3b2f-1fed-433d-85c7-dc1998404596 2025-08-11 01:40:12,841 - DEBUG - Requesting: http://10.0.0.49:8082/virtual-network/ebf405a4-d3f3-4bd9-a2a6-b1df816e0759 2025-08-11 01:40:12,853 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2025-08-11 01:40:12,853 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2025-08-11 01:40:12,861 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-08-11 01:40:12,869 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2025-08-11 01:40:12,869 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2025-08-11 01:40:12,872 - DEBUG - Ingress ctest-nginx-ingress-72941227 not present 2025-08-11 01:40:12,873 - INFO - Creating Ingress ctest-nginx-ingress-72941227 2025-08-11 01:40:12,880 - INFO - Created Ingress ctest-nginx-ingress-72941227 2025-08-11 01:40:12,884 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-72941227 2025-08-11 01:40:15,889 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-72941227 2025-08-11 01:40:18,894 - DEBUG - For Ingress ctest-nginx-ingress-72941227, Cluster IP: 10.47.255.248, External IPs ['81.101.17.252'] 2025-08-11 01:40:18,895 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-72941227 2025-08-11 01:40:18,895 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=5025f0ab-9656-4c41-8f08-493da654ca5d 2025-08-11 01:40:18,900 - INFO - Ingress ctest-nginx-ingress-72941227 with uuid 5025f0ab-9656-4c41-8f08-493da654ca5d found in kube manager 2025-08-11 01:40:18,901 - INFO - Ingress ctest-nginx-ingress-72941227 verification passed 2025-08-11 01:40:18,904 - DEBUG - Pod ctest-busybox-pod-84461075 not present 2025-08-11 01:40:18,904 - INFO - Creating Pod ctest-busybox-pod-84461075 2025-08-11 01:40:18,920 - DEBUG - Pod : ctest-busybox-pod-84461075 UUID is 08d0084e-1891-431d-82f0-e7253d7e4756 2025-08-11 01:40:18,926 - DEBUG - Pod ctest-nginx-pod-46069662 not in running state.Currently in Pending 2025-08-11 01:40:23,935 - DEBUG - Pod ctest-nginx-pod-46069662 not in running state.Currently in Pending 2025-08-11 01:40:28,942 - INFO - Pod ctest-nginx-pod-46069662 is in running state.Got IP 10.47.255.250 2025-08-11 01:40:28,994 - DEBUG - Pod ctest-nginx-pod-46069662 has vmi 2057bd7c-7654-11f0-a5e8-0242cd4ba357 2025-08-11 01:40:28,994 - INFO - Verified pod ctest-nginx-pod-46069662 in contrail-api 2025-08-11 01:40:28,994 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-08-11 01:40:29,007 - DEBUG - VMI 2057bd7c-7654-11f0-a5e8-0242cd4ba357 is active in agent 10.0.0.49 2025-08-11 01:40:29,007 - INFO - Verified Pod ctest-nginx-pod-46069662 in agent 10.0.0.49 2025-08-11 01:40:29,007 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_VirtualMachineDatabaseList?x=8392971d-a26f-4a1a-9e8a-43212a1158d6 2025-08-11 01:40:29,011 - INFO - Pod ctest-nginx-pod-46069662 with uuid 8392971d-a26f-4a1a-9e8a-43212a1158d6 found in kube manager 2025-08-11 01:40:29,011 - INFO - Pod ctest-nginx-pod-46069662 verification passed 2025-08-11 01:40:29,089 - DEBUG - [Pod ctest-nginx-pod-46069662] Cmd: echo ctest-nginx-pod-46069662 > /usr/share/nginx/html/index.html, Output: 2025-08-11 01:40:29,096 - INFO - Pod ctest-nginx-pod-96563660 is in running state.Got IP 10.47.255.249 2025-08-11 01:40:29,148 - DEBUG - Pod ctest-nginx-pod-96563660 has vmi 21040eec-7654-11f0-a5e8-0242cd4ba357 2025-08-11 01:40:29,148 - INFO - Verified pod ctest-nginx-pod-96563660 in contrail-api 2025-08-11 01:40:29,148 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-08-11 01:40:29,158 - DEBUG - VMI 21040eec-7654-11f0-a5e8-0242cd4ba357 is active in agent 10.0.0.49 2025-08-11 01:40:29,158 - INFO - Verified Pod ctest-nginx-pod-96563660 in agent 10.0.0.49 2025-08-11 01:40:29,158 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_VirtualMachineDatabaseList?x=810e1ce0-ff3b-4649-a3de-66a129e43818 2025-08-11 01:40:29,162 - INFO - Pod ctest-nginx-pod-96563660 with uuid 810e1ce0-ff3b-4649-a3de-66a129e43818 found in kube manager 2025-08-11 01:40:29,162 - INFO - Pod ctest-nginx-pod-96563660 verification passed 2025-08-11 01:40:29,230 - DEBUG - [Pod ctest-nginx-pod-96563660] Cmd: echo ctest-nginx-pod-96563660 > /usr/share/nginx/html/index.html, Output: 2025-08-11 01:40:29,236 - INFO - Pod ctest-busybox-pod-84461075 is in running state.Got IP 10.47.255.247 2025-08-11 01:40:29,290 - DEBUG - Pod ctest-busybox-pod-84461075 has vmi 24c6add2-7654-11f0-a5e8-0242cd4ba357 2025-08-11 01:40:29,290 - INFO - Verified pod ctest-busybox-pod-84461075 in contrail-api 2025-08-11 01:40:29,290 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-08-11 01:40:29,299 - DEBUG - VMI 24c6add2-7654-11f0-a5e8-0242cd4ba357 is active in agent 10.0.0.49 2025-08-11 01:40:29,300 - INFO - Verified Pod ctest-busybox-pod-84461075 in agent 10.0.0.49 2025-08-11 01:40:29,300 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_VirtualMachineDatabaseList?x=08d0084e-1891-431d-82f0-e7253d7e4756 2025-08-11 01:40:29,304 - INFO - Pod ctest-busybox-pod-84461075 with uuid 08d0084e-1891-431d-82f0-e7253d7e4756 found in kube manager 2025-08-11 01:40:29,304 - INFO - Pod ctest-busybox-pod-84461075 verification passed 2025-08-11 01:40:29,376 - DEBUG - [Pod ctest-busybox-pod-84461075] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% ctest-nginx-pod-96563660 |********************************| 25 0:00:00 ETA written to stdout 2025-08-11 01:40:29,376 - DEBUG - [Pod Pod ctest-busybox-pod-84461075] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-08-11 01:40:29,376 - DEBUG - [Pod Pod ctest-busybox-pod-84461075] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% ctest-nginx-pod-96563660 |********************************| 25 0:00:00 ETA written to stdout 2025-08-11 01:40:29,441 - DEBUG - [Pod ctest-busybox-pod-84461075] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-46069662 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-08-11 01:40:29,441 - DEBUG - [Pod Pod ctest-busybox-pod-84461075] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-08-11 01:40:29,441 - DEBUG - [Pod Pod ctest-busybox-pod-84461075] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-46069662 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-08-11 01:40:29,441 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-46069662': 1, 'ctest-nginx-pod-96563660': 1} 2025-08-11 01:40:29,441 - WARNING - Nothing to delete parallely 2025-08-11 01:40:29,442 - INFO - Deleting pod default:ctest-busybox-pod-84461075 2025-08-11 01:40:29,451 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=08d0084e-1891-431d-82f0-e7253d7e4756 2025-08-11 01:40:29,458 - WARNING - Pod uuid 08d0084e-1891-431d-82f0-e7253d7e4756 is still seen in agent 10.0.0.49 VM list 2025-08-11 01:40:34,458 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=08d0084e-1891-431d-82f0-e7253d7e4756 2025-08-11 01:40:34,464 - WARNING - Pod uuid 08d0084e-1891-431d-82f0-e7253d7e4756 is still seen in agent 10.0.0.49 VM list 2025-08-11 01:40:39,464 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=08d0084e-1891-431d-82f0-e7253d7e4756 2025-08-11 01:40:39,471 - WARNING - Pod uuid 08d0084e-1891-431d-82f0-e7253d7e4756 is still seen in agent 10.0.0.49 VM list 2025-08-11 01:40:44,472 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=08d0084e-1891-431d-82f0-e7253d7e4756 2025-08-11 01:40:44,480 - WARNING - Pod uuid 08d0084e-1891-431d-82f0-e7253d7e4756 is still seen in agent 10.0.0.49 VM list 2025-08-11 01:40:49,481 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=08d0084e-1891-431d-82f0-e7253d7e4756 2025-08-11 01:40:49,487 - WARNING - Pod uuid 08d0084e-1891-431d-82f0-e7253d7e4756 is still seen in agent 10.0.0.49 VM list 2025-08-11 01:40:54,488 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=08d0084e-1891-431d-82f0-e7253d7e4756 2025-08-11 01:40:54,493 - WARNING - Pod uuid 08d0084e-1891-431d-82f0-e7253d7e4756 is still seen in agent 10.0.0.49 VM list 2025-08-11 01:40:59,494 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=08d0084e-1891-431d-82f0-e7253d7e4756 2025-08-11 01:40:59,502 - WARNING - Pod uuid 08d0084e-1891-431d-82f0-e7253d7e4756 is still seen in agent 10.0.0.49 VM list 2025-08-11 01:41:04,503 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=08d0084e-1891-431d-82f0-e7253d7e4756 2025-08-11 01:41:04,509 - DEBUG - Pod 08d0084e-1891-431d-82f0-e7253d7e4756 is not in agent 10.0.0.49 VM list 2025-08-11 01:41:04,509 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:24c6add2-7654-11f0-a5e8-0242cd4ba357 2025-08-11 01:41:04,514 - DEBUG - VMI 24c6add2-7654-11f0-a5e8-0242cd4ba357 is removed from agent 10.0.0.49 2025-08-11 01:41:04,514 - INFO - Verified that pod ctest-busybox-pod-84461075 is removed in agent 2025-08-11 01:41:04,514 - INFO - Deleting Ingress : ctest-nginx-ingress-72941227 2025-08-11 01:41:04,520 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=5025f0ab-9656-4c41-8f08-493da654ca5d 2025-08-11 01:41:04,524 - ERROR - Ingress ctest-nginx-ingress-72941227 with uuid 5025f0ab-9656-4c41-8f08-493da654ca5d still found in kube manager 2025-08-11 01:41:05,525 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=5025f0ab-9656-4c41-8f08-493da654ca5d 2025-08-11 01:41:05,532 - ERROR - Ingress ctest-nginx-ingress-72941227 with uuid 5025f0ab-9656-4c41-8f08-493da654ca5d still found in kube manager 2025-08-11 01:41:06,532 - DEBUG - Requesting: http://10.0.0.49:8108/Snh_LoadbalancerDatabaseList?x=5025f0ab-9656-4c41-8f08-493da654ca5d 2025-08-11 01:41:06,537 - INFO - Ingress ctest-nginx-ingress-72941227 with uuid 5025f0ab-9656-4c41-8f08-493da654ca5d deleted successfully from kube manager 2025-08-11 01:41:06,537 - INFO - Verifications on Ingress %s cleanup passed 2025-08-11 01:41:06,537 - INFO - Deleting the FIP pool __fip_pool_public__ 2025-08-11 01:41:06,634 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-08-11 01:41:06,640 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2025-08-11 01:41:06,640 - INFO - Deleting VN __public__ 2025-08-11 01:41:06,725 - DEBUG - Requesting: http://10.0.0.49:8082/routing-instance/159b6f27-5c13-4b97-a14a-09e908aa4c5d 2025-08-11 01:41:06,732 - DEBUG - Response Code: 404 2025-08-11 01:41:06,732 - DEBUG - Requesting: http://10.0.0.49:8082/domains 2025-08-11 01:41:06,736 - DEBUG - Requesting: http://10.0.0.49:8082/domain/85e18e77-68eb-48c5-86da-0dd561703453 2025-08-11 01:41:06,743 - DEBUG - Requesting: http://10.0.0.49:8082/project/eb9d3b2f-1fed-433d-85c7-dc1998404596 2025-08-11 01:41:06,775 - INFO - Validated that VN __public__ is not found in API Server 2025-08-11 01:41:06,775 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2025-08-11 01:41:06,781 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-11 01:41:06,787 - DEBUG - VN __public__ is not present in Agent 10.0.0.49 2025-08-11 01:41:06,787 - INFO - Validated that VN __public__ is not in any agent 2025-08-11 01:41:06,788 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-11 01:41:06,795 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-08-11 01:41:06,800 - DEBUG - Requesting: http://10.0.0.49:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-08-11 01:41:06,806 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2025-08-11 01:41:06,806 - INFO - Deleting pod default:ctest-nginx-pod-96563660 2025-08-11 01:41:06,817 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=810e1ce0-ff3b-4649-a3de-66a129e43818 2025-08-11 01:41:06,827 - WARNING - Pod uuid 810e1ce0-ff3b-4649-a3de-66a129e43818 is still seen in agent 10.0.0.49 VM list 2025-08-11 01:41:11,828 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=810e1ce0-ff3b-4649-a3de-66a129e43818 2025-08-11 01:41:11,833 - DEBUG - Pod 810e1ce0-ff3b-4649-a3de-66a129e43818 is not in agent 10.0.0.49 VM list 2025-08-11 01:41:11,833 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:21040eec-7654-11f0-a5e8-0242cd4ba357 2025-08-11 01:41:11,838 - DEBUG - VMI 21040eec-7654-11f0-a5e8-0242cd4ba357 is removed from agent 10.0.0.49 2025-08-11 01:41:11,838 - INFO - Verified that pod ctest-nginx-pod-96563660 is removed in agent 2025-08-11 01:41:11,838 - INFO - Deleting pod default:ctest-nginx-pod-46069662 2025-08-11 01:41:11,847 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=8392971d-a26f-4a1a-9e8a-43212a1158d6 2025-08-11 01:41:11,856 - WARNING - Pod uuid 8392971d-a26f-4a1a-9e8a-43212a1158d6 is still seen in agent 10.0.0.49 VM list 2025-08-11 01:41:16,857 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=8392971d-a26f-4a1a-9e8a-43212a1158d6 2025-08-11 01:41:16,862 - DEBUG - Pod 8392971d-a26f-4a1a-9e8a-43212a1158d6 is not in agent 10.0.0.49 VM list 2025-08-11 01:41:16,862 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:2057bd7c-7654-11f0-a5e8-0242cd4ba357 2025-08-11 01:41:16,867 - DEBUG - VMI 2057bd7c-7654-11f0-a5e8-0242cd4ba357 is removed from agent 10.0.0.49 2025-08-11 01:41:16,867 - INFO - Verified that pod ctest-nginx-pod-46069662 is removed in agent 2025-08-11 01:41:16,867 - INFO - Deleting service : ctest-nginx-svc-25695655 2025-08-11 01:41:17,342 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 01:41:17,349 - DEBUG - No XMPP flaps were noticed during the test