2026-03-12 01:37:08,407 - DEBUG - [10.0.0.50]: Running cmd : hostname 2026-03-12 01:37:08,688 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-366-1 2026-03-12 01:37:08,688 - DEBUG - [10.0.0.50]: Running cmd : hostname -f 2026-03-12 01:37:08,811 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local 2026-03-12 01:37:08,811 - DEBUG - [10.0.0.50]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-03-12 01:37:09,018 - DEBUG - Output : NAMES contrail_test_KGIaZSqwP k8s_autoscaler_dns-autoscaler-595558c478-ljcd5_kube-system_25b86f99-3ae4-4435-9286-dd2d2b88a2cc_0 k8s_coredns_coredns-77f7cc69db-mgj49_kube-system_855c2412-27d8-44d6-93f3-baa3b362ff3b_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-x688z_kube-system_5c640a63-0df6-40f5-8f12-10f5e257661b_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-x688z_kube-system_5c640a63-0df6-40f5-8f12-10f5e257661b_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-x688z_kube-system_5c640a63-0df6-40f5-8f12-10f5e257661b_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-mcjz9_kube-system_d1e30a13-b5dc-4022-a556-ceba7920080e_0 k8s_opensdn-controller-control-named_opensdn-controller-control-mcjz9_kube-system_d1e30a13-b5dc-4022-a556-ceba7920080e_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-cdrzz_kube-system_74d6c7ec-a4ad-4572-8d6b-267f8e00fd4a_1 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-xx5q4_kube-system_fa572b8c-0470-4492-954b-023e175e1bb1_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-cdrzz_kube-system_74d6c7ec-a4ad-4572-8d6b-267f8e00fd4a_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-cdrzz_kube-system_74d6c7ec-a4ad-4572-8d6b-267f8e00fd4a_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-x688z_kube-system_5c640a63-0df6-40f5-8f12-10f5e257661b_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-tg8ms_kube-system_85c79adc-4665-45ff-8e2e-e6d0a32bbeee_0 k8s_opensdn-analytics-collector_opensdn-analytics-tg8ms_kube-system_85c79adc-4665-45ff-8e2e-e6d0a32bbeee_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-mcjz9_kube-system_d1e30a13-b5dc-4022-a556-ceba7920080e_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-kc8rr_kube-system_afebaca2-e5bd-4800-b1ea-5b8b21373faf_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-xx5q4_kube-system_fa572b8c-0470-4492-954b-023e175e1bb1_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-v57bw_kube-system_afddcd32-c246-4ca0-a70b-708669ed9def_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-v57bw_kube-system_afddcd32-c246-4ca0-a70b-708669ed9def_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-xvjq7_kube-system_d577f13b-3d36-4ed3-8801-bfc34d2ed9da_0 k8s_opensdn-controller-config-api_opensdn-controller-config-x688z_kube-system_5c640a63-0df6-40f5-8f12-10f5e257661b_0 k8s_opensdn-analytics-api_opensdn-analytics-tg8ms_kube-system_85c79adc-4665-45ff-8e2e-e6d0a32bbeee_0 k8s_opensdn-controller-control_opensdn-controller-control-mcjz9_kube-system_d1e30a13-b5dc-4022-a556-ceba7920080e_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-xx5q4_kube-system_fa572b8c-0470-4492-954b-023e175e1bb1_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-kc8rr_kube-system_afebaca2-e5bd-4800-b1ea-5b8b21373faf_0 k8s_kafka_opensdn-analytics-alarm-v57bw_kube-system_afddcd32-c246-4ca0-a70b-708669ed9def_0 k8s_opensdn-agent-nodemgr_opensdn-agent-nhpfs_kube-system_07cf33a5-e31f-4827-980c-5c06b49e9eb7_0 k8s_rabbitmq_rabbitmq-n9s2f_kube-system_1e10f98c-864d-4617-b764-d3d58bf550d6_0 k8s_redis_redis-pdswl_kube-system_d10c7b11-1d60-49ec-8383-4d226a8beb90_0 k8s_config-zookeeper_config-zookeeper-82rsw_kube-system_63f901a8-d0df-4d66-ac8b-6b454427fee3_0 k8s_opensdn-configdb_opensdn-configdb-xvjq7_kube-system_d577f13b-3d36-4ed3-8801-bfc34d2ed9da_0 k8s_opensdn-vrouter-agent_opensdn-agent-nhpfs_kube-system_07cf33a5-e31f-4827-980c-5c06b49e9eb7_0 k8s_opensdn-kube-manager_opensdn-kube-manager-9zhk4_kube-system_22ac5809-8ef8-4973-9584-de2ad2e1db57_0 k8s_kube-proxy_kube-proxy-8m8q8_kube-system_d0d11ed3-d882-4039-91c7-2d037385ad61_0 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-366-1_kube-system_189be2c32cda7f5784c7f2fc0371516f_1 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-366-1_kube-system_89417c55fa139a85a9ac2211be56a5fa_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-366-1_kube-system_8e550b1a8f7764022cb02186805e8e35_0 2026-03-12 01:37:09,019 - DEBUG - [10.0.0.50]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-03-12 01:37:09,134 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.1/32 10.96.0.3/32 10.0.0.50/24 2026-03-12 01:37:09,135 - DEBUG - [10.0.0.50]: Running cmd : getent hosts 10.0.0.50 | head -n 1 | awk '{print $2}' 2026-03-12 01:37:09,258 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-366-1.cluster.local 2026-03-12 01:37:09,258 - DEBUG - [10.0.0.50]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-03-12 01:37:09,365 - DEBUG - Output : 10.0.0.50/24 2026-03-12 01:37:09,757 - INFO - Using existing project ['default-domain', 'k8s-default'](aec59006-3a83-454c-9cff-a44de759eca6) 2026-03-12 01:37:09,910 - INFO - Using existing project ['default-domain', 'k8s-default'](aec59006-3a83-454c-9cff-a44de759eca6) 2026-03-12 01:37:10,054 - INFO - ================================================================================ 2026-03-12 01:37:10,054 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-03-12 01:37:10,054 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-03-12 01:37:10,326 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq? 2026-03-12 01:37:10,331 - DEBUG - Nothing to compare xmpp stats {'10.0.0.50': {'10.0.0.50': '2'}} with 2026-03-12 01:37:10,331 - INFO - Initial checks done. Running the testcase now 2026-03-12 01:37:10,332 - INFO - 2026-03-12 01:37:10,336 - INFO - Namespace default already exists 2026-03-12 01:37:10,337 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-9zhk4_kube-system_22ac5809-8ef8-4973-9584-de2ad2e1db57_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-03-12 01:37:10,525 - DEBUG - Output : cluster_project={} 2026-03-12 01:37:10,628 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-03-12 01:37:10,628 - DEBUG - Project uuid in Contrail is aec59006-3a83-454c-9cff-a44de759eca6 2026-03-12 01:37:10,628 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-9zhk4_kube-system_22ac5809-8ef8-4973-9584-de2ad2e1db57_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-03-12 01:37:10,787 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-03-12 01:37:10,788 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_NamespaceDatabaseList?namespace_uuid=72d99741-f3f1-42e4-9b92-d63d4b6f68e0 2026-03-12 01:37:10,793 - INFO - Namespace default with uuid 72d99741-f3f1-42e4-9b92-d63d4b6f68e0 found in kube manager 2026-03-12 01:37:10,793 - INFO - Namespace default verification passed 2026-03-12 01:37:10,797 - DEBUG - Service ctest-nginx-svc-11154161 not present 2026-03-12 01:37:10,798 - INFO - Creating service ctest-nginx-svc-11154161 2026-03-12 01:37:10,816 - INFO - Created Service ctest-nginx-svc-11154161 2026-03-12 01:37:10,821 - DEBUG - Pod ctest-nginx-pod-68265563 not present 2026-03-12 01:37:10,821 - INFO - Creating Pod ctest-nginx-pod-68265563 2026-03-12 01:37:10,856 - DEBUG - Pod : ctest-nginx-pod-68265563 UUID is aafe7713-0ed6-4eca-9b99-bd1bcad1bad8 2026-03-12 01:37:10,861 - DEBUG - Pod ctest-nginx-pod-84001039 not present 2026-03-12 01:37:10,861 - INFO - Creating Pod ctest-nginx-pod-84001039 2026-03-12 01:37:10,890 - DEBUG - Pod : ctest-nginx-pod-84001039 UUID is e1d9747a-93b7-4613-b660-5c3259e3ee76 2026-03-12 01:37:11,182 - INFO - Created VN __public__, UUID :2dcdebc0-c690-4b2c-a7b2-83cd14397001 2026-03-12 01:37:11,435 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-12 01:37:11,443 - DEBUG - Requesting: http://10.0.0.50:8082/domain/889d6a42-eb1e-48e6-940d-44bfcdda031f 2026-03-12 01:37:11,454 - DEBUG - Requesting: http://10.0.0.50:8082/project/aec59006-3a83-454c-9cff-a44de759eca6 2026-03-12 01:37:11,614 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/2dcdebc0-c690-4b2c-a7b2-83cd14397001 2026-03-12 01:37:11,629 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/2dcdebc0-c690-4b2c-a7b2-83cd14397001 2026-03-12 01:37:11,641 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/772450eb-9352-4008-9a34-c4ce3b54e01c 2026-03-12 01:37:11,648 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/772450eb-9352-4008-9a34-c4ce3b54e01c 2026-03-12 01:37:11,655 - DEBUG - Route Targets: [] 2026-03-12 01:37:11,655 - DEBUG - RT names not yet present for VN __public__ 2026-03-12 01:37:16,655 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-12 01:37:16,661 - DEBUG - Requesting: http://10.0.0.50:8082/domain/889d6a42-eb1e-48e6-940d-44bfcdda031f 2026-03-12 01:37:16,668 - DEBUG - Requesting: http://10.0.0.50:8082/project/aec59006-3a83-454c-9cff-a44de759eca6 2026-03-12 01:37:16,703 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/2dcdebc0-c690-4b2c-a7b2-83cd14397001 2026-03-12 01:37:16,713 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/2dcdebc0-c690-4b2c-a7b2-83cd14397001 2026-03-12 01:37:16,722 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/772450eb-9352-4008-9a34-c4ce3b54e01c 2026-03-12 01:37:16,741 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/772450eb-9352-4008-9a34-c4ce3b54e01c 2026-03-12 01:37:16,749 - DEBUG - Requesting: http://10.0.0.50:8082/route-target/b2cecd31-8d15-415d-9ab9-2495100e1a60 2026-03-12 01:37:16,755 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-12 01:37:16,755 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/2dcdebc0-c690-4b2c-a7b2-83cd14397001 2026-03-12 01:37:16,765 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/772450eb-9352-4008-9a34-c4ce3b54e01c 2026-03-12 01:37:16,782 - INFO - Verified VN network id 11 for VN 2dcdebc0-c690-4b2c-a7b2-83cd14397001 2026-03-12 01:37:16,782 - INFO - Verifications in API Server for VN __public__ passed 2026-03-12 01:37:16,782 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/2dcdebc0-c690-4b2c-a7b2-83cd14397001 2026-03-12 01:37:16,791 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/772450eb-9352-4008-9a34-c4ce3b54e01c 2026-03-12 01:37:16,799 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/772450eb-9352-4008-9a34-c4ce3b54e01c 2026-03-12 01:37:16,807 - DEBUG - Requesting: http://10.0.0.50:8082/route-target/b2cecd31-8d15-415d-9ab9-2495100e1a60 2026-03-12 01:37:16,814 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-03-12 01:37:16,823 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3300553315130034988', 'uuid-lslong': '12083865667019304961'}, 'enable': 'true', 'created': '2026-03-12T01:37:11', 'last-modified': '2026-03-12T01:37:11', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.501717'} 2026-03-12 01:37:16,823 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-12 01:37:16,832 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-12 01:37:16,832 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-03-12 01:37:16,833 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-03-12 01:37:16,833 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-12 01:37:16,840 - DEBUG - Requesting: http://10.0.0.50:8082/domain/889d6a42-eb1e-48e6-940d-44bfcdda031f 2026-03-12 01:37:16,848 - DEBUG - Requesting: http://10.0.0.50:8082/project/aec59006-3a83-454c-9cff-a44de759eca6 2026-03-12 01:37:16,883 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/2dcdebc0-c690-4b2c-a7b2-83cd14397001 2026-03-12 01:37:16,911 - DEBUG - =>VN __public__ has no policy to be verified 2026-03-12 01:37:16,911 - DEBUG - Verifying the vn in opserver 2026-03-12 01:37:16,911 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.50 2026-03-12 01:37:16,911 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks 2026-03-12 01:37:16,915 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-03-12 01:37:16,915 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-03-12 01:37:16,915 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-12 01:37:16,923 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-12 01:37:16,928 - DEBUG - Do not have enough data to verify VN in agent 2026-03-12 01:37:16,928 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-12 01:37:16,932 - DEBUG - VRF ids for VN __public__: {} 2026-03-12 01:37:17,109 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-03-12 01:37:17,320 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-12 01:37:17,331 - DEBUG - Requesting: http://10.0.0.50:8082/domain/889d6a42-eb1e-48e6-940d-44bfcdda031f 2026-03-12 01:37:17,339 - DEBUG - Requesting: http://10.0.0.50:8082/project/aec59006-3a83-454c-9cff-a44de759eca6 2026-03-12 01:37:17,421 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/2dcdebc0-c690-4b2c-a7b2-83cd14397001 2026-03-12 01:37:17,448 - DEBUG - Requesting: http://10.0.0.50:8082/floating-ip-pool/22cbf8f7-f09f-49df-aaef-21b6cb0b0b9e 2026-03-12 01:37:17,459 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-12 01:37:17,472 - DEBUG - Requesting: http://10.0.0.50:8082/domain/889d6a42-eb1e-48e6-940d-44bfcdda031f 2026-03-12 01:37:17,481 - DEBUG - Requesting: http://10.0.0.50:8082/project/aec59006-3a83-454c-9cff-a44de759eca6 2026-03-12 01:37:17,515 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/2dcdebc0-c690-4b2c-a7b2-83cd14397001 2026-03-12 01:37:17,532 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-03-12 01:37:17,532 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-03-12 01:37:17,540 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-12 01:37:17,544 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-03-12 01:37:17,544 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-03-12 01:37:17,548 - DEBUG - Ingress ctest-nginx-ingress-11372999 not present 2026-03-12 01:37:17,549 - INFO - Creating Ingress ctest-nginx-ingress-11372999 2026-03-12 01:37:17,556 - INFO - Created Ingress ctest-nginx-ingress-11372999 2026-03-12 01:37:17,562 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-11372999 2026-03-12 01:37:20,569 - DEBUG - For Ingress ctest-nginx-ingress-11372999, Cluster IP: 10.47.255.248, External IPs ['165.152.66.252'] 2026-03-12 01:37:20,569 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-11372999 2026-03-12 01:37:20,569 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=dd702d8b-ffbb-4ab7-bf35-da86e0de65da 2026-03-12 01:37:20,602 - INFO - Ingress ctest-nginx-ingress-11372999 with uuid dd702d8b-ffbb-4ab7-bf35-da86e0de65da found in kube manager 2026-03-12 01:37:20,602 - INFO - Ingress ctest-nginx-ingress-11372999 verification passed 2026-03-12 01:37:20,608 - DEBUG - Pod ctest-busybox-pod-24688870 not present 2026-03-12 01:37:20,609 - INFO - Creating Pod ctest-busybox-pod-24688870 2026-03-12 01:37:20,636 - DEBUG - Node for Pod ctest-busybox-pod-24688870 not yet populated 2026-03-12 01:37:23,645 - DEBUG - Pod : ctest-busybox-pod-24688870 UUID is 47242c9c-ba73-4387-882c-05c9c47f8b64 2026-03-12 01:37:23,654 - DEBUG - Pod ctest-nginx-pod-68265563 not in running state.Currently in Pending 2026-03-12 01:37:28,661 - INFO - Pod ctest-nginx-pod-68265563 is in running state.Got IP 10.47.255.250 2026-03-12 01:37:28,720 - DEBUG - Pod ctest-nginx-pod-68265563 has vmi febef54c-1db3-11f1-befe-0242e66778c6 2026-03-12 01:37:28,721 - INFO - Verified pod ctest-nginx-pod-68265563 in contrail-api 2026-03-12 01:37:28,721 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-12 01:37:28,730 - DEBUG - VMI febef54c-1db3-11f1-befe-0242e66778c6 is active in agent 10.0.0.50 2026-03-12 01:37:28,731 - INFO - Verified Pod ctest-nginx-pod-68265563 in agent 10.0.0.50 2026-03-12 01:37:28,731 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=aafe7713-0ed6-4eca-9b99-bd1bcad1bad8 2026-03-12 01:37:28,734 - INFO - Pod ctest-nginx-pod-68265563 with uuid aafe7713-0ed6-4eca-9b99-bd1bcad1bad8 found in kube manager 2026-03-12 01:37:28,734 - INFO - Pod ctest-nginx-pod-68265563 verification passed 2026-03-12 01:37:28,834 - DEBUG - [Pod ctest-nginx-pod-68265563] Cmd: echo ctest-nginx-pod-68265563 > /usr/share/nginx/html/index.html, Output: 2026-03-12 01:37:28,843 - INFO - Pod ctest-nginx-pod-84001039 is in running state.Got IP 10.47.255.249 2026-03-12 01:37:28,895 - DEBUG - Pod ctest-nginx-pod-84001039 has vmi ff6eaece-1db3-11f1-befe-0242e66778c6 2026-03-12 01:37:28,895 - INFO - Verified pod ctest-nginx-pod-84001039 in contrail-api 2026-03-12 01:37:28,896 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-12 01:37:28,907 - DEBUG - VMI ff6eaece-1db3-11f1-befe-0242e66778c6 is active in agent 10.0.0.50 2026-03-12 01:37:28,907 - INFO - Verified Pod ctest-nginx-pod-84001039 in agent 10.0.0.50 2026-03-12 01:37:28,907 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=e1d9747a-93b7-4613-b660-5c3259e3ee76 2026-03-12 01:37:28,912 - INFO - Pod ctest-nginx-pod-84001039 with uuid e1d9747a-93b7-4613-b660-5c3259e3ee76 found in kube manager 2026-03-12 01:37:28,912 - INFO - Pod ctest-nginx-pod-84001039 verification passed 2026-03-12 01:37:29,010 - DEBUG - [Pod ctest-nginx-pod-84001039] Cmd: echo ctest-nginx-pod-84001039 > /usr/share/nginx/html/index.html, Output: 2026-03-12 01:37:29,019 - DEBUG - Pod ctest-busybox-pod-24688870 not in running state.Currently in Pending 2026-03-12 01:37:34,027 - INFO - Pod ctest-busybox-pod-24688870 is in running state.Got IP 10.47.255.247 2026-03-12 01:37:34,083 - DEBUG - Pod ctest-busybox-pod-24688870 has vmi 04de1c28-1db4-11f1-befe-0242e66778c6 2026-03-12 01:37:34,083 - INFO - Verified pod ctest-busybox-pod-24688870 in contrail-api 2026-03-12 01:37:34,084 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-12 01:37:34,092 - DEBUG - VMI 04de1c28-1db4-11f1-befe-0242e66778c6 is active in agent 10.0.0.50 2026-03-12 01:37:34,092 - INFO - Verified Pod ctest-busybox-pod-24688870 in agent 10.0.0.50 2026-03-12 01:37:34,092 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=47242c9c-ba73-4387-882c-05c9c47f8b64 2026-03-12 01:37:34,096 - INFO - Pod ctest-busybox-pod-24688870 with uuid 47242c9c-ba73-4387-882c-05c9c47f8b64 found in kube manager 2026-03-12 01:37:34,096 - INFO - Pod ctest-busybox-pod-24688870 verification passed 2026-03-12 01:37:34,177 - DEBUG - [Pod ctest-busybox-pod-24688870] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-84001039 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-12 01:37:34,178 - DEBUG - [Pod Pod ctest-busybox-pod-24688870] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-12 01:37:34,178 - DEBUG - [Pod Pod ctest-busybox-pod-24688870] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-84001039 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-12 01:37:34,268 - DEBUG - [Pod ctest-busybox-pod-24688870] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-68265563 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-12 01:37:34,268 - DEBUG - [Pod Pod ctest-busybox-pod-24688870] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-12 01:37:34,268 - DEBUG - [Pod Pod ctest-busybox-pod-24688870] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-68265563 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-12 01:37:34,268 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-68265563': 1, 'ctest-nginx-pod-84001039': 1} 2026-03-12 01:37:34,269 - WARNING - Nothing to delete parallely 2026-03-12 01:37:34,269 - INFO - Deleting pod default:ctest-busybox-pod-24688870 2026-03-12 01:37:34,283 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=47242c9c-ba73-4387-882c-05c9c47f8b64 2026-03-12 01:37:34,289 - WARNING - Pod uuid 47242c9c-ba73-4387-882c-05c9c47f8b64 is still seen in agent 10.0.0.50 VM list 2026-03-12 01:37:39,289 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=47242c9c-ba73-4387-882c-05c9c47f8b64 2026-03-12 01:37:39,296 - WARNING - Pod uuid 47242c9c-ba73-4387-882c-05c9c47f8b64 is still seen in agent 10.0.0.50 VM list 2026-03-12 01:37:44,297 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=47242c9c-ba73-4387-882c-05c9c47f8b64 2026-03-12 01:37:44,303 - WARNING - Pod uuid 47242c9c-ba73-4387-882c-05c9c47f8b64 is still seen in agent 10.0.0.50 VM list 2026-03-12 01:37:49,304 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=47242c9c-ba73-4387-882c-05c9c47f8b64 2026-03-12 01:37:49,310 - WARNING - Pod uuid 47242c9c-ba73-4387-882c-05c9c47f8b64 is still seen in agent 10.0.0.50 VM list 2026-03-12 01:37:54,311 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=47242c9c-ba73-4387-882c-05c9c47f8b64 2026-03-12 01:37:54,318 - WARNING - Pod uuid 47242c9c-ba73-4387-882c-05c9c47f8b64 is still seen in agent 10.0.0.50 VM list 2026-03-12 01:37:59,319 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=47242c9c-ba73-4387-882c-05c9c47f8b64 2026-03-12 01:37:59,324 - WARNING - Pod uuid 47242c9c-ba73-4387-882c-05c9c47f8b64 is still seen in agent 10.0.0.50 VM list 2026-03-12 01:38:04,325 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=47242c9c-ba73-4387-882c-05c9c47f8b64 2026-03-12 01:38:04,332 - WARNING - Pod uuid 47242c9c-ba73-4387-882c-05c9c47f8b64 is still seen in agent 10.0.0.50 VM list 2026-03-12 01:38:09,333 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=47242c9c-ba73-4387-882c-05c9c47f8b64 2026-03-12 01:38:09,340 - DEBUG - Pod 47242c9c-ba73-4387-882c-05c9c47f8b64 is not in agent 10.0.0.50 VM list 2026-03-12 01:38:09,340 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:04de1c28-1db4-11f1-befe-0242e66778c6 2026-03-12 01:38:09,347 - DEBUG - VMI 04de1c28-1db4-11f1-befe-0242e66778c6 is removed from agent 10.0.0.50 2026-03-12 01:38:09,347 - INFO - Verified that pod ctest-busybox-pod-24688870 is removed in agent 2026-03-12 01:38:09,347 - INFO - Deleting Ingress : ctest-nginx-ingress-11372999 2026-03-12 01:38:09,356 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=dd702d8b-ffbb-4ab7-bf35-da86e0de65da 2026-03-12 01:38:09,361 - ERROR - Ingress ctest-nginx-ingress-11372999 with uuid dd702d8b-ffbb-4ab7-bf35-da86e0de65da still found in kube manager 2026-03-12 01:38:10,362 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=dd702d8b-ffbb-4ab7-bf35-da86e0de65da 2026-03-12 01:38:10,366 - ERROR - Ingress ctest-nginx-ingress-11372999 with uuid dd702d8b-ffbb-4ab7-bf35-da86e0de65da still found in kube manager 2026-03-12 01:38:11,367 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=dd702d8b-ffbb-4ab7-bf35-da86e0de65da 2026-03-12 01:38:11,373 - INFO - Ingress ctest-nginx-ingress-11372999 with uuid dd702d8b-ffbb-4ab7-bf35-da86e0de65da deleted successfully from kube manager 2026-03-12 01:38:11,373 - INFO - Verifications on Ingress %s cleanup passed 2026-03-12 01:38:11,373 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-03-12 01:38:11,473 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-12 01:38:11,479 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-03-12 01:38:11,479 - INFO - Deleting VN __public__ 2026-03-12 01:38:11,664 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/772450eb-9352-4008-9a34-c4ce3b54e01c 2026-03-12 01:38:11,679 - DEBUG - Response Code: 404 2026-03-12 01:38:11,679 - DEBUG - Requesting: http://10.0.0.50:8082/domains 2026-03-12 01:38:11,696 - DEBUG - Requesting: http://10.0.0.50:8082/domain/889d6a42-eb1e-48e6-940d-44bfcdda031f 2026-03-12 01:38:11,704 - DEBUG - Requesting: http://10.0.0.50:8082/project/aec59006-3a83-454c-9cff-a44de759eca6 2026-03-12 01:38:11,743 - INFO - Validated that VN __public__ is not found in API Server 2026-03-12 01:38:11,743 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-03-12 01:38:11,750 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-12 01:38:11,756 - DEBUG - VN __public__ is not present in Agent 10.0.0.50 2026-03-12 01:38:11,756 - INFO - Validated that VN __public__ is not in any agent 2026-03-12 01:38:11,756 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-12 01:38:11,763 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-12 01:38:11,774 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-03-12 01:38:11,779 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-03-12 01:38:11,780 - INFO - Deleting pod default:ctest-nginx-pod-84001039 2026-03-12 01:38:11,792 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=e1d9747a-93b7-4613-b660-5c3259e3ee76 2026-03-12 01:38:11,801 - WARNING - Pod uuid e1d9747a-93b7-4613-b660-5c3259e3ee76 is still seen in agent 10.0.0.50 VM list 2026-03-12 01:38:16,802 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=e1d9747a-93b7-4613-b660-5c3259e3ee76 2026-03-12 01:38:16,808 - DEBUG - Pod e1d9747a-93b7-4613-b660-5c3259e3ee76 is not in agent 10.0.0.50 VM list 2026-03-12 01:38:16,808 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ff6eaece-1db3-11f1-befe-0242e66778c6 2026-03-12 01:38:16,812 - DEBUG - VMI ff6eaece-1db3-11f1-befe-0242e66778c6 is removed from agent 10.0.0.50 2026-03-12 01:38:16,812 - INFO - Verified that pod ctest-nginx-pod-84001039 is removed in agent 2026-03-12 01:38:16,813 - INFO - Deleting pod default:ctest-nginx-pod-68265563 2026-03-12 01:38:16,833 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=aafe7713-0ed6-4eca-9b99-bd1bcad1bad8 2026-03-12 01:38:16,839 - WARNING - Pod uuid aafe7713-0ed6-4eca-9b99-bd1bcad1bad8 is still seen in agent 10.0.0.50 VM list 2026-03-12 01:38:21,840 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_VmListReq?uuid=aafe7713-0ed6-4eca-9b99-bd1bcad1bad8 2026-03-12 01:38:21,844 - DEBUG - Pod aafe7713-0ed6-4eca-9b99-bd1bcad1bad8 is not in agent 10.0.0.50 VM list 2026-03-12 01:38:21,844 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:febef54c-1db3-11f1-befe-0242e66778c6 2026-03-12 01:38:21,849 - DEBUG - VMI febef54c-1db3-11f1-befe-0242e66778c6 is removed from agent 10.0.0.50 2026-03-12 01:38:21,849 - INFO - Verified that pod ctest-nginx-pod-68265563 is removed in agent 2026-03-12 01:38:21,849 - INFO - Deleting service : ctest-nginx-svc-11154161 2026-03-12 01:38:22,379 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq? 2026-03-12 01:38:22,384 - DEBUG - No XMPP flaps were noticed during the test