2026-03-19 12:28:59,080 - DEBUG - [10.0.0.38]: Running cmd : hostname 2026-03-19 12:28:59,352 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-379-1 2026-03-19 12:28:59,352 - DEBUG - [10.0.0.38]: Running cmd : hostname -f 2026-03-19 12:28:59,475 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-379-1.cluster.local 2026-03-19 12:28:59,476 - DEBUG - [10.0.0.38]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-03-19 12:28:59,638 - DEBUG - Output : NAMES contrail_test_NsiPQZSDY k8s_coredns_coredns-77f7cc69db-58pzk_kube-system_ddedeb89-dd7f-4ce4-a72e-24a89fa80db2_0 k8s_autoscaler_dns-autoscaler-595558c478-zd47h_kube-system_2b85d826-7b36-4818-89ed-9f66dea20db3_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-w85xw_kube-system_36f41a37-2645-45d0-aa03-a92df1ff9831_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-w85xw_kube-system_36f41a37-2645-45d0-aa03-a92df1ff9831_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-w85xw_kube-system_36f41a37-2645-45d0-aa03-a92df1ff9831_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-npswn_kube-system_a9a90e23-6347-4ce2-8fc7-b37776760629_0 k8s_opensdn-controller-control-named_opensdn-controller-control-npswn_kube-system_a9a90e23-6347-4ce2-8fc7-b37776760629_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-lpnpd_kube-system_78ede369-be2c-438b-ac6e-e15cb3e92f4e_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-mzxx8_kube-system_2a8823b0-244f-4eb0-9e1d-d69412d7b75d_0 k8s_opensdn-analytics-collector_opensdn-analytics-mzxx8_kube-system_2a8823b0-244f-4eb0-9e1d-d69412d7b75d_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-t94zq_kube-system_60da6234-894e-46ac-b353-32fa1ad59951_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-t94zq_kube-system_60da6234-894e-46ac-b353-32fa1ad59951_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-w85xw_kube-system_36f41a37-2645-45d0-aa03-a92df1ff9831_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-67qwk_kube-system_f118ed63-454a-432b-b407-e08382851fdb_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-npswn_kube-system_a9a90e23-6347-4ce2-8fc7-b37776760629_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-xjfwf_kube-system_700c3157-a3b0-45b1-a7d4-cc2877457b9d_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-xjfwf_kube-system_700c3157-a3b0-45b1-a7d4-cc2877457b9d_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-lpnpd_kube-system_78ede369-be2c-438b-ac6e-e15cb3e92f4e_0 k8s_opensdn-analytics-api_opensdn-analytics-mzxx8_kube-system_2a8823b0-244f-4eb0-9e1d-d69412d7b75d_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-t94zq_kube-system_60da6234-894e-46ac-b353-32fa1ad59951_0 k8s_opensdn-controller-config-api_opensdn-controller-config-w85xw_kube-system_36f41a37-2645-45d0-aa03-a92df1ff9831_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-67qwk_kube-system_f118ed63-454a-432b-b407-e08382851fdb_0 k8s_opensdn-controller-control_opensdn-controller-control-npswn_kube-system_a9a90e23-6347-4ce2-8fc7-b37776760629_0 k8s_kafka_opensdn-analytics-alarm-xjfwf_kube-system_700c3157-a3b0-45b1-a7d4-cc2877457b9d_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-s6jjx_kube-system_af4f32bc-083f-466c-bb3c-f23682bd614d_0 k8s_opensdn-configdb_opensdn-configdb-s6jjx_kube-system_af4f32bc-083f-466c-bb3c-f23682bd614d_0 k8s_config-zookeeper_config-zookeeper-mqlcr_kube-system_17a38ef0-5635-4346-b862-fd0337769ccd_0 k8s_redis_redis-p29nk_kube-system_82613ae1-241a-4206-8805-afd1299f835d_0 k8s_rabbitmq_rabbitmq-97c5c_kube-system_d2594701-a36b-4e14-a450-5ae472ef6f1c_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-67qwk_kube-system_f118ed63-454a-432b-b407-e08382851fdb_0 k8s_opensdn-agent-nodemgr_opensdn-agent-pbkb2_kube-system_11197c36-9cae-42b6-8a49-19f7a43b781c_0 k8s_opensdn-vrouter-agent_opensdn-agent-pbkb2_kube-system_11197c36-9cae-42b6-8a49-19f7a43b781c_0 k8s_opensdn-kube-manager_opensdn-kube-manager-plpr8_kube-system_99bca6bc-77fe-48c9-89ed-c7c17b5b0697_0 k8s_kube-proxy_kube-proxy-fbl6m_kube-system_ed3f88b1-fe1b-4d90-9858-530bf63bf509_0 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-379-1_kube-system_6631bab9c7fca6cf9b85cb37085d8a73_1 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-379-1_kube-system_3759db3c4403c55544f1b1557c92cd61_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-379-1_kube-system_3bde7400aaec062e6a66b0ecc907099f_0 2026-03-19 12:28:59,638 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-03-19 12:28:59,758 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.3/32 10.96.0.1/32 10.0.0.38/24 2026-03-19 12:28:59,759 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.0.0.38 | head -n 1 | awk '{print $2}' 2026-03-19 12:28:59,885 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-379-1.cluster.local 2026-03-19 12:28:59,885 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-03-19 12:29:00,007 - DEBUG - Output : 10.0.0.38/24 2026-03-19 12:29:00,374 - INFO - Using existing project ['default-domain', 'k8s-default'](3aff8a55-910b-4999-9fb1-07185515b8b8) 2026-03-19 12:29:00,506 - INFO - Using existing project ['default-domain', 'k8s-default'](3aff8a55-910b-4999-9fb1-07185515b8b8) 2026-03-19 12:29:00,629 - INFO - ================================================================================ 2026-03-19 12:29:00,629 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-03-19 12:29:00,629 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-03-19 12:29:00,919 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_AgentXmppConnectionStatusReq? 2026-03-19 12:29:00,926 - DEBUG - Nothing to compare xmpp stats {'10.0.0.38': {'10.0.0.38': '2'}} with 2026-03-19 12:29:00,926 - INFO - Initial checks done. Running the testcase now 2026-03-19 12:29:00,926 - INFO - 2026-03-19 12:29:00,930 - INFO - Namespace default already exists 2026-03-19 12:29:00,931 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-plpr8_kube-system_99bca6bc-77fe-48c9-89ed-c7c17b5b0697_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-03-19 12:29:01,119 - DEBUG - Output : cluster_project={} 2026-03-19 12:29:01,199 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-03-19 12:29:01,200 - DEBUG - Project uuid in Contrail is 3aff8a55-910b-4999-9fb1-07185515b8b8 2026-03-19 12:29:01,200 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-plpr8_kube-system_99bca6bc-77fe-48c9-89ed-c7c17b5b0697_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-03-19 12:29:01,380 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-03-19 12:29:01,380 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_NamespaceDatabaseList?namespace_uuid=81cba095-84f8-47cf-abb9-05157dd93527 2026-03-19 12:29:01,385 - INFO - Namespace default with uuid 81cba095-84f8-47cf-abb9-05157dd93527 found in kube manager 2026-03-19 12:29:01,385 - INFO - Namespace default verification passed 2026-03-19 12:29:01,390 - DEBUG - Service ctest-nginx-svc-67124680 not present 2026-03-19 12:29:01,390 - INFO - Creating service ctest-nginx-svc-67124680 2026-03-19 12:29:01,402 - INFO - Created Service ctest-nginx-svc-67124680 2026-03-19 12:29:01,406 - DEBUG - Pod ctest-nginx-pod-55128494 not present 2026-03-19 12:29:01,407 - INFO - Creating Pod ctest-nginx-pod-55128494 2026-03-19 12:29:01,462 - DEBUG - Pod : ctest-nginx-pod-55128494 UUID is f2cde5f6-0594-4a41-8615-aece1e8b03e7 2026-03-19 12:29:01,470 - DEBUG - Pod ctest-nginx-pod-22494367 not present 2026-03-19 12:29:01,471 - INFO - Creating Pod ctest-nginx-pod-22494367 2026-03-19 12:29:01,515 - DEBUG - Pod : ctest-nginx-pod-22494367 UUID is 96559bc5-6039-4ea5-beee-8676370aada0 2026-03-19 12:29:02,018 - INFO - Created VN __public__, UUID :6591ba0d-b816-41ea-8325-16432e8ebdf2 2026-03-19 12:29:02,321 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-03-19 12:29:02,334 - DEBUG - Requesting: http://10.0.0.38:8082/domain/652c2dd1-2687-415f-99fc-b3db419fa787 2026-03-19 12:29:02,365 - DEBUG - Requesting: http://10.0.0.38:8082/project/3aff8a55-910b-4999-9fb1-07185515b8b8 2026-03-19 12:29:02,491 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6591ba0d-b816-41ea-8325-16432e8ebdf2 2026-03-19 12:29:02,672 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6591ba0d-b816-41ea-8325-16432e8ebdf2 2026-03-19 12:29:02,734 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/55f286b9-67c6-4e34-8a4d-450a257cc29a 2026-03-19 12:29:02,827 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/55f286b9-67c6-4e34-8a4d-450a257cc29a 2026-03-19 12:29:02,864 - DEBUG - Route Targets: [] 2026-03-19 12:29:02,864 - DEBUG - RT names not yet present for VN __public__ 2026-03-19 12:29:07,865 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-03-19 12:29:07,870 - DEBUG - Requesting: http://10.0.0.38:8082/domain/652c2dd1-2687-415f-99fc-b3db419fa787 2026-03-19 12:29:07,877 - DEBUG - Requesting: http://10.0.0.38:8082/project/3aff8a55-910b-4999-9fb1-07185515b8b8 2026-03-19 12:29:07,919 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6591ba0d-b816-41ea-8325-16432e8ebdf2 2026-03-19 12:29:07,928 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6591ba0d-b816-41ea-8325-16432e8ebdf2 2026-03-19 12:29:07,936 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/55f286b9-67c6-4e34-8a4d-450a257cc29a 2026-03-19 12:29:07,944 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/55f286b9-67c6-4e34-8a4d-450a257cc29a 2026-03-19 12:29:07,951 - DEBUG - Requesting: http://10.0.0.38:8082/route-target/512271ff-0a9a-426f-8e0c-fe173053253b 2026-03-19 12:29:07,957 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-19 12:29:07,957 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6591ba0d-b816-41ea-8325-16432e8ebdf2 2026-03-19 12:29:07,965 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/55f286b9-67c6-4e34-8a4d-450a257cc29a 2026-03-19 12:29:07,978 - INFO - Verified VN network id 11 for VN 6591ba0d-b816-41ea-8325-16432e8ebdf2 2026-03-19 12:29:07,978 - INFO - Verifications in API Server for VN __public__ passed 2026-03-19 12:29:07,978 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6591ba0d-b816-41ea-8325-16432e8ebdf2 2026-03-19 12:29:07,987 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/55f286b9-67c6-4e34-8a4d-450a257cc29a 2026-03-19 12:29:07,993 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/55f286b9-67c6-4e34-8a4d-450a257cc29a 2026-03-19 12:29:08,000 - DEBUG - Requesting: http://10.0.0.38:8082/route-target/512271ff-0a9a-426f-8e0c-fe173053253b 2026-03-19 12:29:08,018 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-03-19 12:29:08,034 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7318835437539574250', 'uuid-lslong': '9449983870906580466'}, 'enable': 'true', 'created': '2026-03-19T12:29:01', 'last-modified': '2026-03-19T12:29:02', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.797489'} 2026-03-19 12:29:08,034 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-19 12:29:08,047 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-19 12:29:08,047 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-03-19 12:29:08,050 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-03-19 12:29:08,051 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-03-19 12:29:08,062 - DEBUG - Requesting: http://10.0.0.38:8082/domain/652c2dd1-2687-415f-99fc-b3db419fa787 2026-03-19 12:29:08,074 - DEBUG - Requesting: http://10.0.0.38:8082/project/3aff8a55-910b-4999-9fb1-07185515b8b8 2026-03-19 12:29:08,115 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6591ba0d-b816-41ea-8325-16432e8ebdf2 2026-03-19 12:29:08,132 - DEBUG - =>VN __public__ has no policy to be verified 2026-03-19 12:29:08,132 - DEBUG - Verifying the vn in opserver 2026-03-19 12:29:08,132 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.38 2026-03-19 12:29:08,132 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2026-03-19 12:29:08,138 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-03-19 12:29:08,138 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-03-19 12:29:08,139 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-19 12:29:08,148 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-19 12:29:08,153 - DEBUG - Do not have enough data to verify VN in agent 2026-03-19 12:29:08,153 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-19 12:29:08,158 - DEBUG - VRF ids for VN __public__: {} 2026-03-19 12:29:08,324 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-03-19 12:29:08,588 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-03-19 12:29:08,596 - DEBUG - Requesting: http://10.0.0.38:8082/domain/652c2dd1-2687-415f-99fc-b3db419fa787 2026-03-19 12:29:08,603 - DEBUG - Requesting: http://10.0.0.38:8082/project/3aff8a55-910b-4999-9fb1-07185515b8b8 2026-03-19 12:29:08,638 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6591ba0d-b816-41ea-8325-16432e8ebdf2 2026-03-19 12:29:08,652 - DEBUG - Requesting: http://10.0.0.38:8082/floating-ip-pool/6dc74517-2b89-4281-ac6a-f608bbb20143 2026-03-19 12:29:08,658 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-03-19 12:29:08,668 - DEBUG - Requesting: http://10.0.0.38:8082/domain/652c2dd1-2687-415f-99fc-b3db419fa787 2026-03-19 12:29:08,676 - DEBUG - Requesting: http://10.0.0.38:8082/project/3aff8a55-910b-4999-9fb1-07185515b8b8 2026-03-19 12:29:08,705 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6591ba0d-b816-41ea-8325-16432e8ebdf2 2026-03-19 12:29:08,727 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-03-19 12:29:08,727 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-03-19 12:29:08,740 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-19 12:29:08,752 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-03-19 12:29:08,752 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-03-19 12:29:08,756 - DEBUG - Ingress ctest-nginx-ingress-79692125 not present 2026-03-19 12:29:08,757 - INFO - Creating Ingress ctest-nginx-ingress-79692125 2026-03-19 12:29:08,770 - INFO - Created Ingress ctest-nginx-ingress-79692125 2026-03-19 12:29:08,775 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-79692125 2026-03-19 12:29:11,781 - DEBUG - For Ingress ctest-nginx-ingress-79692125, Cluster IP: 10.47.255.248, External IPs ['130.48.92.60'] 2026-03-19 12:29:11,781 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-79692125 2026-03-19 12:29:11,781 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=2bcdd33c-abd3-4141-aa7d-d3cd79403b8a 2026-03-19 12:29:11,785 - INFO - Ingress ctest-nginx-ingress-79692125 with uuid 2bcdd33c-abd3-4141-aa7d-d3cd79403b8a found in kube manager 2026-03-19 12:29:11,785 - INFO - Ingress ctest-nginx-ingress-79692125 verification passed 2026-03-19 12:29:11,790 - DEBUG - Pod ctest-busybox-pod-55203030 not present 2026-03-19 12:29:11,790 - INFO - Creating Pod ctest-busybox-pod-55203030 2026-03-19 12:29:11,809 - DEBUG - Pod : ctest-busybox-pod-55203030 UUID is f9d1950c-752b-496d-9d99-5b55c653acd2 2026-03-19 12:29:11,821 - DEBUG - Pod ctest-nginx-pod-55128494 not in running state.Currently in Pending 2026-03-19 12:29:16,831 - DEBUG - Pod ctest-nginx-pod-55128494 not in running state.Currently in Pending 2026-03-19 12:29:21,840 - INFO - Pod ctest-nginx-pod-55128494 is in running state.Got IP 10.47.255.250 2026-03-19 12:29:21,894 - DEBUG - Pod ctest-nginx-pod-55128494 has vmi 37bf70e8-238f-11f1-89c2-02421b3ceccd 2026-03-19 12:29:21,894 - INFO - Verified pod ctest-nginx-pod-55128494 in contrail-api 2026-03-19 12:29:21,894 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-19 12:29:21,904 - DEBUG - VMI 37bf70e8-238f-11f1-89c2-02421b3ceccd is active in agent 10.0.0.38 2026-03-19 12:29:21,904 - INFO - Verified Pod ctest-nginx-pod-55128494 in agent 10.0.0.38 2026-03-19 12:29:21,904 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=f2cde5f6-0594-4a41-8615-aece1e8b03e7 2026-03-19 12:29:21,909 - INFO - Pod ctest-nginx-pod-55128494 with uuid f2cde5f6-0594-4a41-8615-aece1e8b03e7 found in kube manager 2026-03-19 12:29:21,909 - INFO - Pod ctest-nginx-pod-55128494 verification passed 2026-03-19 12:29:22,009 - DEBUG - [Pod ctest-nginx-pod-55128494] Cmd: echo ctest-nginx-pod-55128494 > /usr/share/nginx/html/index.html, Output: 2026-03-19 12:29:22,018 - INFO - Pod ctest-nginx-pod-22494367 is in running state.Got IP 10.47.255.249 2026-03-19 12:29:22,073 - DEBUG - Pod ctest-nginx-pod-22494367 has vmi 388285ce-238f-11f1-89c2-02421b3ceccd 2026-03-19 12:29:22,073 - INFO - Verified pod ctest-nginx-pod-22494367 in contrail-api 2026-03-19 12:29:22,073 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-19 12:29:22,083 - DEBUG - VMI 388285ce-238f-11f1-89c2-02421b3ceccd is active in agent 10.0.0.38 2026-03-19 12:29:22,083 - INFO - Verified Pod ctest-nginx-pod-22494367 in agent 10.0.0.38 2026-03-19 12:29:22,083 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=96559bc5-6039-4ea5-beee-8676370aada0 2026-03-19 12:29:22,088 - INFO - Pod ctest-nginx-pod-22494367 with uuid 96559bc5-6039-4ea5-beee-8676370aada0 found in kube manager 2026-03-19 12:29:22,089 - INFO - Pod ctest-nginx-pod-22494367 verification passed 2026-03-19 12:29:22,165 - DEBUG - [Pod ctest-nginx-pod-22494367] Cmd: echo ctest-nginx-pod-22494367 > /usr/share/nginx/html/index.html, Output: 2026-03-19 12:29:22,171 - INFO - Pod ctest-busybox-pod-55203030 is in running state.Got IP 10.47.255.247 2026-03-19 12:29:22,224 - DEBUG - Pod ctest-busybox-pod-55203030 has vmi 3d89cd0c-238f-11f1-89c2-02421b3ceccd 2026-03-19 12:29:22,225 - INFO - Verified pod ctest-busybox-pod-55203030 in contrail-api 2026-03-19 12:29:22,225 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-19 12:29:22,233 - DEBUG - VMI 3d89cd0c-238f-11f1-89c2-02421b3ceccd is active in agent 10.0.0.38 2026-03-19 12:29:22,233 - INFO - Verified Pod ctest-busybox-pod-55203030 in agent 10.0.0.38 2026-03-19 12:29:22,234 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=f9d1950c-752b-496d-9d99-5b55c653acd2 2026-03-19 12:29:22,238 - INFO - Pod ctest-busybox-pod-55203030 with uuid f9d1950c-752b-496d-9d99-5b55c653acd2 found in kube manager 2026-03-19 12:29:22,238 - INFO - Pod ctest-busybox-pod-55203030 verification passed 2026-03-19 12:29:22,311 - DEBUG - [Pod ctest-busybox-pod-55203030] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-55128494 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-19 12:29:22,311 - DEBUG - [Pod Pod ctest-busybox-pod-55203030] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-19 12:29:22,311 - DEBUG - [Pod Pod ctest-busybox-pod-55203030] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-55128494 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-19 12:29:22,399 - DEBUG - [Pod ctest-busybox-pod-55203030] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-55128494 - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-19 12:29:22,399 - DEBUG - [Pod Pod ctest-busybox-pod-55203030] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-19 12:29:22,399 - DEBUG - [Pod Pod ctest-busybox-pod-55203030] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-55128494 - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-19 12:29:22,470 - DEBUG - [Pod ctest-busybox-pod-55203030] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-55128494 - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-19 12:29:22,471 - DEBUG - [Pod Pod ctest-busybox-pod-55203030] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-19 12:29:22,471 - DEBUG - [Pod Pod ctest-busybox-pod-55203030] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-55128494 - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-19 12:29:22,542 - DEBUG - [Pod ctest-busybox-pod-55203030] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-55128494 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-19 12:29:22,542 - DEBUG - [Pod Pod ctest-busybox-pod-55203030] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-19 12:29:22,543 - DEBUG - [Pod Pod ctest-busybox-pod-55203030] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-55128494 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-19 12:29:22,609 - DEBUG - [Pod ctest-busybox-pod-55203030] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-55128494 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-19 12:29:22,609 - DEBUG - [Pod Pod ctest-busybox-pod-55203030] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-19 12:29:22,609 - DEBUG - [Pod Pod ctest-busybox-pod-55203030] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-55128494 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-19 12:29:22,673 - DEBUG - [Pod ctest-busybox-pod-55203030] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-22494367 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-19 12:29:22,673 - DEBUG - [Pod Pod ctest-busybox-pod-55203030] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-19 12:29:22,673 - DEBUG - [Pod Pod ctest-busybox-pod-55203030] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-22494367 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-19 12:29:22,673 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-55128494': 5, 'ctest-nginx-pod-22494367': 1} 2026-03-19 12:29:22,673 - WARNING - Nothing to delete parallely 2026-03-19 12:29:22,673 - INFO - Deleting pod default:ctest-busybox-pod-55203030 2026-03-19 12:29:22,686 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f9d1950c-752b-496d-9d99-5b55c653acd2 2026-03-19 12:29:22,692 - WARNING - Pod uuid f9d1950c-752b-496d-9d99-5b55c653acd2 is still seen in agent 10.0.0.38 VM list 2026-03-19 12:29:27,693 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f9d1950c-752b-496d-9d99-5b55c653acd2 2026-03-19 12:29:27,698 - WARNING - Pod uuid f9d1950c-752b-496d-9d99-5b55c653acd2 is still seen in agent 10.0.0.38 VM list 2026-03-19 12:29:32,699 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f9d1950c-752b-496d-9d99-5b55c653acd2 2026-03-19 12:29:32,706 - WARNING - Pod uuid f9d1950c-752b-496d-9d99-5b55c653acd2 is still seen in agent 10.0.0.38 VM list 2026-03-19 12:29:37,707 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f9d1950c-752b-496d-9d99-5b55c653acd2 2026-03-19 12:29:37,712 - WARNING - Pod uuid f9d1950c-752b-496d-9d99-5b55c653acd2 is still seen in agent 10.0.0.38 VM list 2026-03-19 12:29:42,713 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f9d1950c-752b-496d-9d99-5b55c653acd2 2026-03-19 12:29:42,718 - WARNING - Pod uuid f9d1950c-752b-496d-9d99-5b55c653acd2 is still seen in agent 10.0.0.38 VM list 2026-03-19 12:29:47,719 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f9d1950c-752b-496d-9d99-5b55c653acd2 2026-03-19 12:29:47,724 - WARNING - Pod uuid f9d1950c-752b-496d-9d99-5b55c653acd2 is still seen in agent 10.0.0.38 VM list 2026-03-19 12:29:52,725 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f9d1950c-752b-496d-9d99-5b55c653acd2 2026-03-19 12:29:52,730 - WARNING - Pod uuid f9d1950c-752b-496d-9d99-5b55c653acd2 is still seen in agent 10.0.0.38 VM list 2026-03-19 12:29:57,731 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f9d1950c-752b-496d-9d99-5b55c653acd2 2026-03-19 12:29:57,737 - DEBUG - Pod f9d1950c-752b-496d-9d99-5b55c653acd2 is not in agent 10.0.0.38 VM list 2026-03-19 12:29:57,738 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:3d89cd0c-238f-11f1-89c2-02421b3ceccd 2026-03-19 12:29:57,742 - DEBUG - VMI 3d89cd0c-238f-11f1-89c2-02421b3ceccd is removed from agent 10.0.0.38 2026-03-19 12:29:57,742 - INFO - Verified that pod ctest-busybox-pod-55203030 is removed in agent 2026-03-19 12:29:57,742 - INFO - Deleting Ingress : ctest-nginx-ingress-79692125 2026-03-19 12:29:57,749 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=2bcdd33c-abd3-4141-aa7d-d3cd79403b8a 2026-03-19 12:29:57,753 - ERROR - Ingress ctest-nginx-ingress-79692125 with uuid 2bcdd33c-abd3-4141-aa7d-d3cd79403b8a still found in kube manager 2026-03-19 12:29:58,754 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=2bcdd33c-abd3-4141-aa7d-d3cd79403b8a 2026-03-19 12:29:58,759 - ERROR - Ingress ctest-nginx-ingress-79692125 with uuid 2bcdd33c-abd3-4141-aa7d-d3cd79403b8a still found in kube manager 2026-03-19 12:29:59,760 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=2bcdd33c-abd3-4141-aa7d-d3cd79403b8a 2026-03-19 12:29:59,764 - INFO - Ingress ctest-nginx-ingress-79692125 with uuid 2bcdd33c-abd3-4141-aa7d-d3cd79403b8a deleted successfully from kube manager 2026-03-19 12:29:59,764 - INFO - Verifications on Ingress %s cleanup passed 2026-03-19 12:29:59,764 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-03-19 12:29:59,980 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-19 12:29:59,990 - WARNING - Control-node ifmap object for FIP pool __fip_pool_public__ , VN __public__ is found! 2026-03-19 12:30:04,992 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-19 12:30:04,999 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-03-19 12:30:04,999 - INFO - Deleting VN __public__ 2026-03-19 12:30:05,229 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/55f286b9-67c6-4e34-8a4d-450a257cc29a 2026-03-19 12:30:05,237 - DEBUG - Response Code: 404 2026-03-19 12:30:05,238 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-03-19 12:30:05,245 - DEBUG - Requesting: http://10.0.0.38:8082/domain/652c2dd1-2687-415f-99fc-b3db419fa787 2026-03-19 12:30:05,254 - DEBUG - Requesting: http://10.0.0.38:8082/project/3aff8a55-910b-4999-9fb1-07185515b8b8 2026-03-19 12:30:05,327 - INFO - Validated that VN __public__ is not found in API Server 2026-03-19 12:30:05,327 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-03-19 12:30:05,334 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-19 12:30:05,341 - DEBUG - VN __public__ is not present in Agent 10.0.0.38 2026-03-19 12:30:05,341 - INFO - Validated that VN __public__ is not in any agent 2026-03-19 12:30:05,341 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-19 12:30:05,348 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-19 12:30:05,354 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-03-19 12:30:05,361 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-03-19 12:30:05,362 - INFO - Deleting pod default:ctest-nginx-pod-22494367 2026-03-19 12:30:05,381 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=96559bc5-6039-4ea5-beee-8676370aada0 2026-03-19 12:30:05,392 - WARNING - Pod uuid 96559bc5-6039-4ea5-beee-8676370aada0 is still seen in agent 10.0.0.38 VM list 2026-03-19 12:30:10,393 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=96559bc5-6039-4ea5-beee-8676370aada0 2026-03-19 12:30:10,402 - DEBUG - Pod 96559bc5-6039-4ea5-beee-8676370aada0 is not in agent 10.0.0.38 VM list 2026-03-19 12:30:10,402 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:388285ce-238f-11f1-89c2-02421b3ceccd 2026-03-19 12:30:10,416 - DEBUG - VMI 388285ce-238f-11f1-89c2-02421b3ceccd is removed from agent 10.0.0.38 2026-03-19 12:30:10,416 - INFO - Verified that pod ctest-nginx-pod-22494367 is removed in agent 2026-03-19 12:30:10,417 - INFO - Deleting pod default:ctest-nginx-pod-55128494 2026-03-19 12:30:10,438 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f2cde5f6-0594-4a41-8615-aece1e8b03e7 2026-03-19 12:30:10,449 - WARNING - Pod uuid f2cde5f6-0594-4a41-8615-aece1e8b03e7 is still seen in agent 10.0.0.38 VM list 2026-03-19 12:30:15,449 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=f2cde5f6-0594-4a41-8615-aece1e8b03e7 2026-03-19 12:30:15,455 - DEBUG - Pod f2cde5f6-0594-4a41-8615-aece1e8b03e7 is not in agent 10.0.0.38 VM list 2026-03-19 12:30:15,455 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:37bf70e8-238f-11f1-89c2-02421b3ceccd 2026-03-19 12:30:15,461 - DEBUG - VMI 37bf70e8-238f-11f1-89c2-02421b3ceccd is removed from agent 10.0.0.38 2026-03-19 12:30:15,461 - INFO - Verified that pod ctest-nginx-pod-55128494 is removed in agent 2026-03-19 12:30:15,462 - INFO - Deleting service : ctest-nginx-svc-67124680 2026-03-19 12:30:16,061 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_AgentXmppConnectionStatusReq? 2026-03-19 12:30:16,067 - DEBUG - No XMPP flaps were noticed during the test