2025-05-22 01:40:02,596 - DEBUG - [10.0.0.57]: Running cmd : hostname 2025-05-22 01:40:02,878 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-190-1 2025-05-22 01:40:02,878 - DEBUG - [10.0.0.57]: Running cmd : hostname -f 2025-05-22 01:40:03,015 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-190-1.cluster.local 2025-05-22 01:40:03,015 - DEBUG - [10.0.0.57]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-05-22 01:40:03,234 - DEBUG - Output : NAMES contrail_test_VVOQbzAhd k8s_autoscaler_dns-autoscaler-595558c478-4vfbc_kube-system_5e1fbedd-e935-4017-a4e8-59d49488ceae_0 k8s_coredns_coredns-77f7cc69db-bd69b_kube-system_e08937ff-06c0-42f0-8b7f-efc4f94944df_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-ms59c_kube-system_01dc405d-b37e-4c9d-af2d-4a3f77c07e4e_1 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-ms59c_kube-system_01dc405d-b37e-4c9d-af2d-4a3f77c07e4e_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-ms59c_kube-system_01dc405d-b37e-4c9d-af2d-4a3f77c07e4e_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-g8572_kube-system_f559a3b0-ecc9-4aaa-82f1-9672526d29f1_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-cptg5_kube-system_f7721e4c-244a-44e4-bfe4-fa36ac12dcdb_0 k8s_opensdn-controller-control-named_opensdn-controller-control-cptg5_kube-system_f7721e4c-244a-44e4-bfe4-fa36ac12dcdb_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-5l4zh_kube-system_2e7cc668-6f0d-41cf-aad3-a216a857b96b_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-5l4zh_kube-system_2e7cc668-6f0d-41cf-aad3-a216a857b96b_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-g8572_kube-system_f559a3b0-ecc9-4aaa-82f1-9672526d29f1_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-qqjbd_kube-system_7a13d7e9-b19f-494e-ac93-a3e43ae91c0f_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-qqjbd_kube-system_7a13d7e9-b19f-494e-ac93-a3e43ae91c0f_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-cptg5_kube-system_f7721e4c-244a-44e4-bfe4-fa36ac12dcdb_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-tfjhf_kube-system_bb19f065-c179-44e9-9673-3207da6a79d4_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-ms59c_kube-system_01dc405d-b37e-4c9d-af2d-4a3f77c07e4e_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-nt5jv_kube-system_841c6838-be07-4f3b-ab29-a3a08b084e20_0 k8s_opensdn-analytics-collector_opensdn-analytics-nt5jv_kube-system_841c6838-be07-4f3b-ab29-a3a08b084e20_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-g8572_kube-system_f559a3b0-ecc9-4aaa-82f1-9672526d29f1_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-5l4zh_kube-system_2e7cc668-6f0d-41cf-aad3-a216a857b96b_0 k8s_kafka_opensdn-analytics-alarm-qqjbd_kube-system_7a13d7e9-b19f-494e-ac93-a3e43ae91c0f_0 k8s_opensdn-controller-control_opensdn-controller-control-cptg5_kube-system_f7721e4c-244a-44e4-bfe4-fa36ac12dcdb_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-tfjhf_kube-system_bb19f065-c179-44e9-9673-3207da6a79d4_0 k8s_opensdn-controller-config-api_opensdn-controller-config-ms59c_kube-system_01dc405d-b37e-4c9d-af2d-4a3f77c07e4e_0 k8s_opensdn-analytics-api_opensdn-analytics-nt5jv_kube-system_841c6838-be07-4f3b-ab29-a3a08b084e20_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-lk9xw_kube-system_285e88b0-eb10-4165-b0b5-e4ea4d8b0503_0 k8s_opensdn-configdb_opensdn-configdb-lk9xw_kube-system_285e88b0-eb10-4165-b0b5-e4ea4d8b0503_0 k8s_redis_redis-dkbg9_kube-system_26e15f49-a040-443e-99cf-40f05d6b995e_0 k8s_rabbitmq_rabbitmq-xf6gx_kube-system_10fb0768-0f3b-418b-b693-91c5bcc81fab_0 k8s_config-zookeeper_config-zookeeper-ftqz2_kube-system_59da689d-99e9-4d9b-9975-bd21b2f64bc2_0 k8s_opensdn-agent-nodemgr_opensdn-agent-x95c7_kube-system_9a673576-7221-42dc-9944-d893149b71fd_0 k8s_opensdn-vrouter-agent_opensdn-agent-x95c7_kube-system_9a673576-7221-42dc-9944-d893149b71fd_0 k8s_opensdn-kube-manager_opensdn-kube-manager-dtckz_kube-system_b9e2885f-a1f7-479e-94cc-147c738fdb0d_0 k8s_kube-proxy_kube-proxy-hjv8p_kube-system_dd13a70a-aef2-458a-89fd-1ed983a1c5ef_0 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-190-1_kube-system_6e4cb9191a6d65869124471612b66135_1 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-190-1_kube-system_f50bbc9eb474398218b267e153e3126c_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-190-1_kube-system_c841b94e8030b8e74d33b122ae1db4d0_0 2025-05-22 01:40:03,235 - DEBUG - [10.0.0.57]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}' 2025-05-22 01:40:03,371 - DEBUG - Output : 172.17.0.1/16 10.0.0.57/24 2025-05-22 01:40:03,371 - DEBUG - [10.0.0.57]: Running cmd : getent hosts 10.0.0.57 | head -n 1 | awk '{print $2}' 2025-05-22 01:40:03,516 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-190-1.cluster.local 2025-05-22 01:40:03,516 - DEBUG - [10.0.0.57]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}' 2025-05-22 01:40:03,646 - DEBUG - Output : 10.0.0.57/24 2025-05-22 01:40:04,024 - INFO - Using existing project ['default-domain', 'k8s-default'](96a82623-7b73-432f-bc31-9696045c237f) 2025-05-22 01:40:04,197 - INFO - Using existing project ['default-domain', 'k8s-default'](96a82623-7b73-432f-bc31-9696045c237f) 2025-05-22 01:40:04,355 - INFO - ================================================================================ 2025-05-22 01:40:04,355 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-05-22 01:40:04,356 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-05-22 01:40:04,637 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_AgentXmppConnectionStatusReq? 2025-05-22 01:40:04,645 - DEBUG - Nothing to compare xmpp stats {'10.0.0.57': {'10.0.0.57': '2'}} with 2025-05-22 01:40:04,645 - INFO - Initial checks done. Running the testcase now 2025-05-22 01:40:04,645 - INFO - 2025-05-22 01:40:04,650 - INFO - Namespace default already exists 2025-05-22 01:40:04,650 - DEBUG - [10.0.0.57]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-dtckz_kube-system_b9e2885f-a1f7-479e-94cc-147c738fdb0d_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-05-22 01:40:04,856 - DEBUG - Output : cluster_project={} 2025-05-22 01:40:04,939 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-05-22 01:40:04,939 - DEBUG - Project uuid in Contrail is 96a82623-7b73-432f-bc31-9696045c237f 2025-05-22 01:40:04,940 - DEBUG - [10.0.0.57]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-dtckz_kube-system_b9e2885f-a1f7-479e-94cc-147c738fdb0d_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-05-22 01:40:05,110 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2025-05-22 01:40:05,111 - DEBUG - Requesting: http://10.0.0.57:8108/Snh_NamespaceDatabaseList?namespace_uuid=362fea92-324b-4cdf-87d8-d801304cb9d7 2025-05-22 01:40:05,116 - INFO - Namespace default with uuid 362fea92-324b-4cdf-87d8-d801304cb9d7 found in kube manager 2025-05-22 01:40:05,116 - INFO - Namespace default verification passed 2025-05-22 01:40:05,120 - DEBUG - Service ctest-nginx-svc-44623582 not present 2025-05-22 01:40:05,120 - INFO - Creating service ctest-nginx-svc-44623582 2025-05-22 01:40:05,136 - INFO - Created Service ctest-nginx-svc-44623582 2025-05-22 01:40:05,147 - DEBUG - Pod ctest-nginx-pod-70236957 not present 2025-05-22 01:40:05,148 - INFO - Creating Pod ctest-nginx-pod-70236957 2025-05-22 01:40:05,185 - DEBUG - Pod : ctest-nginx-pod-70236957 UUID is 8669643c-f0f8-4dc4-9671-3fdd45e17a5c 2025-05-22 01:40:05,191 - DEBUG - Pod ctest-nginx-pod-52820126 not present 2025-05-22 01:40:05,192 - INFO - Creating Pod ctest-nginx-pod-52820126 2025-05-22 01:40:05,217 - DEBUG - Pod : ctest-nginx-pod-52820126 UUID is 5eb13a76-b40b-4fd3-b26e-63648250adcd 2025-05-22 01:40:05,516 - INFO - Created VN __public__, UUID :2fa2c6e6-b870-4577-bdbe-d37f8c62799a 2025-05-22 01:40:05,781 - DEBUG - Requesting: http://10.0.0.57:8082/domains 2025-05-22 01:40:05,796 - DEBUG - Requesting: http://10.0.0.57:8082/domain/a3449e88-8ead-4acd-92d2-af6754a87945 2025-05-22 01:40:05,838 - DEBUG - Requesting: http://10.0.0.57:8082/project/96a82623-7b73-432f-bc31-9696045c237f 2025-05-22 01:40:06,053 - DEBUG - Requesting: http://10.0.0.57:8082/virtual-network/2fa2c6e6-b870-4577-bdbe-d37f8c62799a 2025-05-22 01:40:06,079 - DEBUG - Requesting: http://10.0.0.57:8082/virtual-network/2fa2c6e6-b870-4577-bdbe-d37f8c62799a 2025-05-22 01:40:06,101 - DEBUG - Requesting: http://10.0.0.57:8082/routing-instance/860821e8-0ca5-4fab-bf3c-be438dbda853 2025-05-22 01:40:06,118 - DEBUG - Requesting: http://10.0.0.57:8082/routing-instance/860821e8-0ca5-4fab-bf3c-be438dbda853 2025-05-22 01:40:06,128 - DEBUG - Route Targets: [] 2025-05-22 01:40:06,129 - DEBUG - RT names not yet present for VN __public__ 2025-05-22 01:40:11,129 - DEBUG - Requesting: http://10.0.0.57:8082/domains 2025-05-22 01:40:11,138 - DEBUG - Requesting: http://10.0.0.57:8082/domain/a3449e88-8ead-4acd-92d2-af6754a87945 2025-05-22 01:40:11,164 - DEBUG - Requesting: http://10.0.0.57:8082/project/96a82623-7b73-432f-bc31-9696045c237f 2025-05-22 01:40:11,220 - DEBUG - Requesting: http://10.0.0.57:8082/virtual-network/2fa2c6e6-b870-4577-bdbe-d37f8c62799a 2025-05-22 01:40:11,241 - DEBUG - Requesting: http://10.0.0.57:8082/virtual-network/2fa2c6e6-b870-4577-bdbe-d37f8c62799a 2025-05-22 01:40:11,259 - DEBUG - Requesting: http://10.0.0.57:8082/routing-instance/860821e8-0ca5-4fab-bf3c-be438dbda853 2025-05-22 01:40:11,271 - DEBUG - Requesting: http://10.0.0.57:8082/routing-instance/860821e8-0ca5-4fab-bf3c-be438dbda853 2025-05-22 01:40:11,280 - DEBUG - Requesting: http://10.0.0.57:8082/route-target/deb027d6-58d8-4e34-aea4-4e9de3c458aa 2025-05-22 01:40:11,288 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-05-22 01:40:11,288 - DEBUG - Requesting: http://10.0.0.57:8082/virtual-network/2fa2c6e6-b870-4577-bdbe-d37f8c62799a 2025-05-22 01:40:11,299 - DEBUG - Requesting: http://10.0.0.57:8082/routing-instance/860821e8-0ca5-4fab-bf3c-be438dbda853 2025-05-22 01:40:11,316 - INFO - Verified VN network id 12 for VN 2fa2c6e6-b870-4577-bdbe-d37f8c62799a 2025-05-22 01:40:11,316 - INFO - Verifications in API Server for VN __public__ passed 2025-05-22 01:40:11,316 - DEBUG - Requesting: http://10.0.0.57:8082/virtual-network/2fa2c6e6-b870-4577-bdbe-d37f8c62799a 2025-05-22 01:40:11,327 - DEBUG - Requesting: http://10.0.0.57:8082/routing-instance/860821e8-0ca5-4fab-bf3c-be438dbda853 2025-05-22 01:40:11,335 - DEBUG - Requesting: http://10.0.0.57:8082/routing-instance/860821e8-0ca5-4fab-bf3c-be438dbda853 2025-05-22 01:40:11,344 - DEBUG - Requesting: http://10.0.0.57:8082/route-target/deb027d6-58d8-4e34-aea4-4e9de3c458aa 2025-05-22 01:40:11,351 - DEBUG - Requesting: http://10.0.0.57:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-05-22 01:40:11,357 - DEBUG - Control-node 10.0.0.57 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3432524560248882551', 'uuid-lslong': '13672598063512975770'}, 'enable': 'true', 'created': '2025-05-22T01:40:05', 'last-modified': '2025-05-22T01:40:05', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.695112'} 2025-05-22 01:40:11,357 - DEBUG - Requesting: http://10.0.0.57:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-05-22 01:40:11,365 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-05-22 01:40:11,365 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2025-05-22 01:40:11,366 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2025-05-22 01:40:11,366 - DEBUG - Requesting: http://10.0.0.57:8082/domains 2025-05-22 01:40:11,373 - DEBUG - Requesting: http://10.0.0.57:8082/domain/a3449e88-8ead-4acd-92d2-af6754a87945 2025-05-22 01:40:11,382 - DEBUG - Requesting: http://10.0.0.57:8082/project/96a82623-7b73-432f-bc31-9696045c237f 2025-05-22 01:40:11,426 - DEBUG - Requesting: http://10.0.0.57:8082/virtual-network/2fa2c6e6-b870-4577-bdbe-d37f8c62799a 2025-05-22 01:40:11,440 - DEBUG - =>VN __public__ has no policy to be verified 2025-05-22 01:40:11,440 - DEBUG - Verifying the vn in opserver 2025-05-22 01:40:11,440 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.57 2025-05-22 01:40:11,440 - DEBUG - Requesting: http://10.0.0.57:8081/analytics/uves/virtual-networks 2025-05-22 01:40:11,447 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.57:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2025-05-22 01:40:11,447 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2025-05-22 01:40:11,447 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-05-22 01:40:11,455 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-05-22 01:40:11,462 - DEBUG - Do not have enough data to verify VN in agent 2025-05-22 01:40:11,462 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-05-22 01:40:11,468 - DEBUG - VRF ids for VN __public__: {} 2025-05-22 01:40:11,642 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2025-05-22 01:40:11,873 - DEBUG - Requesting: http://10.0.0.57:8082/domains 2025-05-22 01:40:11,880 - DEBUG - Requesting: http://10.0.0.57:8082/domain/a3449e88-8ead-4acd-92d2-af6754a87945 2025-05-22 01:40:11,890 - DEBUG - Requesting: http://10.0.0.57:8082/project/96a82623-7b73-432f-bc31-9696045c237f 2025-05-22 01:40:11,937 - DEBUG - Requesting: http://10.0.0.57:8082/virtual-network/2fa2c6e6-b870-4577-bdbe-d37f8c62799a 2025-05-22 01:40:11,950 - DEBUG - Requesting: http://10.0.0.57:8082/floating-ip-pool/ccf93754-4d4e-4a91-9fd7-6afc6394ac56 2025-05-22 01:40:11,958 - DEBUG - Requesting: http://10.0.0.57:8082/domains 2025-05-22 01:40:11,963 - DEBUG - Requesting: http://10.0.0.57:8082/domain/a3449e88-8ead-4acd-92d2-af6754a87945 2025-05-22 01:40:11,972 - DEBUG - Requesting: http://10.0.0.57:8082/project/96a82623-7b73-432f-bc31-9696045c237f 2025-05-22 01:40:12,012 - DEBUG - Requesting: http://10.0.0.57:8082/virtual-network/2fa2c6e6-b870-4577-bdbe-d37f8c62799a 2025-05-22 01:40:12,026 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2025-05-22 01:40:12,026 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2025-05-22 01:40:12,035 - DEBUG - Requesting: http://10.0.0.57:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-05-22 01:40:12,041 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2025-05-22 01:40:12,041 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2025-05-22 01:40:12,045 - DEBUG - Ingress ctest-nginx-ingress-70071627 not present 2025-05-22 01:40:12,046 - INFO - Creating Ingress ctest-nginx-ingress-70071627 2025-05-22 01:40:12,052 - INFO - Created Ingress ctest-nginx-ingress-70071627 2025-05-22 01:40:12,057 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-70071627 2025-05-22 01:40:15,068 - DEBUG - For Ingress ctest-nginx-ingress-70071627, Cluster IP: 10.47.255.248, External IPs ['1.3.116.188'] 2025-05-22 01:40:15,068 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-70071627 2025-05-22 01:40:15,068 - DEBUG - Requesting: http://10.0.0.57:8108/Snh_LoadbalancerDatabaseList?x=248549b6-e3af-4c2e-82b1-5038605f5c71 2025-05-22 01:40:15,080 - INFO - Ingress ctest-nginx-ingress-70071627 with uuid 248549b6-e3af-4c2e-82b1-5038605f5c71 found in kube manager 2025-05-22 01:40:15,080 - INFO - Ingress ctest-nginx-ingress-70071627 verification passed 2025-05-22 01:40:15,087 - DEBUG - Pod ctest-busybox-pod-72152160 not present 2025-05-22 01:40:15,087 - INFO - Creating Pod ctest-busybox-pod-72152160 2025-05-22 01:40:15,142 - DEBUG - Pod : ctest-busybox-pod-72152160 UUID is a9492bf5-a46d-4a4c-8f0b-b36300652337 2025-05-22 01:40:15,158 - DEBUG - Pod ctest-nginx-pod-70236957 not in running state.Currently in Pending 2025-05-22 01:40:20,166 - DEBUG - Pod ctest-nginx-pod-70236957 not in running state.Currently in Pending 2025-05-22 01:40:25,174 - INFO - Pod ctest-nginx-pod-70236957 is in running state.Got IP 10.47.255.250 2025-05-22 01:40:25,262 - DEBUG - Pod ctest-nginx-pod-70236957 has vmi b145b096-36ad-11f0-b112-024242daea04 2025-05-22 01:40:25,262 - INFO - Verified pod ctest-nginx-pod-70236957 in contrail-api 2025-05-22 01:40:25,263 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-05-22 01:40:25,275 - DEBUG - VMI b145b096-36ad-11f0-b112-024242daea04 is active in agent 10.0.0.57 2025-05-22 01:40:25,275 - INFO - Verified Pod ctest-nginx-pod-70236957 in agent 10.0.0.57 2025-05-22 01:40:25,275 - DEBUG - Requesting: http://10.0.0.57:8108/Snh_VirtualMachineDatabaseList?x=8669643c-f0f8-4dc4-9671-3fdd45e17a5c 2025-05-22 01:40:25,280 - INFO - Pod ctest-nginx-pod-70236957 with uuid 8669643c-f0f8-4dc4-9671-3fdd45e17a5c found in kube manager 2025-05-22 01:40:25,280 - INFO - Pod ctest-nginx-pod-70236957 verification passed 2025-05-22 01:40:25,420 - DEBUG - [Pod ctest-nginx-pod-70236957] Cmd: echo ctest-nginx-pod-70236957 > /usr/share/nginx/html/index.html, Output: 2025-05-22 01:40:25,427 - INFO - Pod ctest-nginx-pod-52820126 is in running state.Got IP 10.47.255.249 2025-05-22 01:40:25,517 - DEBUG - Pod ctest-nginx-pod-52820126 has vmi b20281a8-36ad-11f0-b112-024242daea04 2025-05-22 01:40:25,517 - INFO - Verified pod ctest-nginx-pod-52820126 in contrail-api 2025-05-22 01:40:25,518 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-05-22 01:40:25,530 - DEBUG - VMI b20281a8-36ad-11f0-b112-024242daea04 is active in agent 10.0.0.57 2025-05-22 01:40:25,531 - INFO - Verified Pod ctest-nginx-pod-52820126 in agent 10.0.0.57 2025-05-22 01:40:25,531 - DEBUG - Requesting: http://10.0.0.57:8108/Snh_VirtualMachineDatabaseList?x=5eb13a76-b40b-4fd3-b26e-63648250adcd 2025-05-22 01:40:25,541 - INFO - Pod ctest-nginx-pod-52820126 with uuid 5eb13a76-b40b-4fd3-b26e-63648250adcd found in kube manager 2025-05-22 01:40:25,541 - INFO - Pod ctest-nginx-pod-52820126 verification passed 2025-05-22 01:40:25,656 - DEBUG - [Pod ctest-nginx-pod-52820126] Cmd: echo ctest-nginx-pod-52820126 > /usr/share/nginx/html/index.html, Output: 2025-05-22 01:40:25,667 - INFO - Pod ctest-busybox-pod-72152160 is in running state.Got IP 10.47.255.247 2025-05-22 01:40:25,742 - DEBUG - Pod ctest-busybox-pod-72152160 has vmi b7ecfbfc-36ad-11f0-b112-024242daea04 2025-05-22 01:40:25,743 - INFO - Verified pod ctest-busybox-pod-72152160 in contrail-api 2025-05-22 01:40:25,743 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-05-22 01:40:25,758 - DEBUG - VMI b7ecfbfc-36ad-11f0-b112-024242daea04 is active in agent 10.0.0.57 2025-05-22 01:40:25,758 - INFO - Verified Pod ctest-busybox-pod-72152160 in agent 10.0.0.57 2025-05-22 01:40:25,758 - DEBUG - Requesting: http://10.0.0.57:8108/Snh_VirtualMachineDatabaseList?x=a9492bf5-a46d-4a4c-8f0b-b36300652337 2025-05-22 01:40:25,765 - INFO - Pod ctest-busybox-pod-72152160 with uuid a9492bf5-a46d-4a4c-8f0b-b36300652337 found in kube manager 2025-05-22 01:40:25,765 - INFO - Pod ctest-busybox-pod-72152160 verification passed 2025-05-22 01:40:26,921 - DEBUG - [Pod ctest-busybox-pod-72152160] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-70236957 - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-05-22 01:40:26,921 - DEBUG - [Pod Pod ctest-busybox-pod-72152160] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-05-22 01:40:26,921 - DEBUG - [Pod Pod ctest-busybox-pod-72152160] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-70236957 - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-05-22 01:40:27,105 - DEBUG - [Pod ctest-busybox-pod-72152160] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-70236957 Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-05-22 01:40:27,105 - DEBUG - [Pod Pod ctest-busybox-pod-72152160] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-05-22 01:40:27,105 - DEBUG - [Pod Pod ctest-busybox-pod-72152160] Cmd output: ctest-nginx-pod-70236957 Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-05-22 01:40:27,222 - DEBUG - [Pod ctest-busybox-pod-72152160] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-70236957 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-05-22 01:40:27,223 - DEBUG - [Pod Pod ctest-busybox-pod-72152160] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-05-22 01:40:27,223 - DEBUG - [Pod Pod ctest-busybox-pod-72152160] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-70236957 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-05-22 01:40:27,305 - DEBUG - [Pod ctest-busybox-pod-72152160] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-52820126 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-05-22 01:40:27,305 - DEBUG - [Pod Pod ctest-busybox-pod-72152160] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2025-05-22 01:40:27,305 - DEBUG - [Pod Pod ctest-busybox-pod-72152160] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-52820126 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-05-22 01:40:27,305 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-70236957': 3, 'ctest-nginx-pod-52820126': 1} 2025-05-22 01:40:27,305 - WARNING - Nothing to delete parallely 2025-05-22 01:40:27,305 - INFO - Deleting pod default:ctest-busybox-pod-72152160 2025-05-22 01:40:27,320 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=a9492bf5-a46d-4a4c-8f0b-b36300652337 2025-05-22 01:40:27,331 - WARNING - Pod uuid a9492bf5-a46d-4a4c-8f0b-b36300652337 is still seen in agent 10.0.0.57 VM list 2025-05-22 01:40:32,332 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=a9492bf5-a46d-4a4c-8f0b-b36300652337 2025-05-22 01:40:32,341 - WARNING - Pod uuid a9492bf5-a46d-4a4c-8f0b-b36300652337 is still seen in agent 10.0.0.57 VM list 2025-05-22 01:40:37,342 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=a9492bf5-a46d-4a4c-8f0b-b36300652337 2025-05-22 01:40:37,350 - WARNING - Pod uuid a9492bf5-a46d-4a4c-8f0b-b36300652337 is still seen in agent 10.0.0.57 VM list 2025-05-22 01:40:42,352 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=a9492bf5-a46d-4a4c-8f0b-b36300652337 2025-05-22 01:40:42,359 - WARNING - Pod uuid a9492bf5-a46d-4a4c-8f0b-b36300652337 is still seen in agent 10.0.0.57 VM list 2025-05-22 01:40:47,360 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=a9492bf5-a46d-4a4c-8f0b-b36300652337 2025-05-22 01:40:47,367 - WARNING - Pod uuid a9492bf5-a46d-4a4c-8f0b-b36300652337 is still seen in agent 10.0.0.57 VM list 2025-05-22 01:40:52,368 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=a9492bf5-a46d-4a4c-8f0b-b36300652337 2025-05-22 01:40:52,374 - WARNING - Pod uuid a9492bf5-a46d-4a4c-8f0b-b36300652337 is still seen in agent 10.0.0.57 VM list 2025-05-22 01:40:57,376 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=a9492bf5-a46d-4a4c-8f0b-b36300652337 2025-05-22 01:40:57,382 - WARNING - Pod uuid a9492bf5-a46d-4a4c-8f0b-b36300652337 is still seen in agent 10.0.0.57 VM list 2025-05-22 01:41:02,384 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=a9492bf5-a46d-4a4c-8f0b-b36300652337 2025-05-22 01:41:02,391 - DEBUG - Pod a9492bf5-a46d-4a4c-8f0b-b36300652337 is not in agent 10.0.0.57 VM list 2025-05-22 01:41:02,391 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b7ecfbfc-36ad-11f0-b112-024242daea04 2025-05-22 01:41:02,397 - DEBUG - VMI b7ecfbfc-36ad-11f0-b112-024242daea04 is removed from agent 10.0.0.57 2025-05-22 01:41:02,397 - INFO - Verified that pod ctest-busybox-pod-72152160 is removed in agent 2025-05-22 01:41:02,398 - INFO - Deleting Ingress : ctest-nginx-ingress-70071627 2025-05-22 01:41:02,404 - DEBUG - Requesting: http://10.0.0.57:8108/Snh_LoadbalancerDatabaseList?x=248549b6-e3af-4c2e-82b1-5038605f5c71 2025-05-22 01:41:02,409 - ERROR - Ingress ctest-nginx-ingress-70071627 with uuid 248549b6-e3af-4c2e-82b1-5038605f5c71 still found in kube manager 2025-05-22 01:41:03,410 - DEBUG - Requesting: http://10.0.0.57:8108/Snh_LoadbalancerDatabaseList?x=248549b6-e3af-4c2e-82b1-5038605f5c71 2025-05-22 01:41:03,415 - ERROR - Ingress ctest-nginx-ingress-70071627 with uuid 248549b6-e3af-4c2e-82b1-5038605f5c71 still found in kube manager 2025-05-22 01:41:04,416 - DEBUG - Requesting: http://10.0.0.57:8108/Snh_LoadbalancerDatabaseList?x=248549b6-e3af-4c2e-82b1-5038605f5c71 2025-05-22 01:41:04,423 - INFO - Ingress ctest-nginx-ingress-70071627 with uuid 248549b6-e3af-4c2e-82b1-5038605f5c71 deleted successfully from kube manager 2025-05-22 01:41:04,423 - INFO - Verifications on Ingress %s cleanup passed 2025-05-22 01:41:04,423 - INFO - Deleting the FIP pool __fip_pool_public__ 2025-05-22 01:41:04,825 - DEBUG - Requesting: http://10.0.0.57:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-05-22 01:41:04,840 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2025-05-22 01:41:04,840 - INFO - Deleting VN __public__ 2025-05-22 01:41:05,071 - DEBUG - Requesting: http://10.0.0.57:8082/routing-instance/860821e8-0ca5-4fab-bf3c-be438dbda853 2025-05-22 01:41:05,092 - DEBUG - Response Code: 404 2025-05-22 01:41:05,092 - DEBUG - Requesting: http://10.0.0.57:8082/domains 2025-05-22 01:41:05,100 - DEBUG - Requesting: http://10.0.0.57:8082/domain/a3449e88-8ead-4acd-92d2-af6754a87945 2025-05-22 01:41:05,108 - DEBUG - Requesting: http://10.0.0.57:8082/project/96a82623-7b73-432f-bc31-9696045c237f 2025-05-22 01:41:05,161 - INFO - Validated that VN __public__ is not found in API Server 2025-05-22 01:41:05,161 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2025-05-22 01:41:05,172 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-05-22 01:41:05,179 - DEBUG - VN __public__ is not present in Agent 10.0.0.57 2025-05-22 01:41:05,179 - INFO - Validated that VN __public__ is not in any agent 2025-05-22 01:41:05,179 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-05-22 01:41:05,186 - DEBUG - Requesting: http://10.0.0.57:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-05-22 01:41:05,193 - DEBUG - Requesting: http://10.0.0.57:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-05-22 01:41:05,200 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2025-05-22 01:41:05,200 - INFO - Deleting pod default:ctest-nginx-pod-52820126 2025-05-22 01:41:05,212 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=5eb13a76-b40b-4fd3-b26e-63648250adcd 2025-05-22 01:41:05,226 - WARNING - Pod uuid 5eb13a76-b40b-4fd3-b26e-63648250adcd is still seen in agent 10.0.0.57 VM list 2025-05-22 01:41:10,227 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=5eb13a76-b40b-4fd3-b26e-63648250adcd 2025-05-22 01:41:10,234 - DEBUG - Pod 5eb13a76-b40b-4fd3-b26e-63648250adcd is not in agent 10.0.0.57 VM list 2025-05-22 01:41:10,234 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b20281a8-36ad-11f0-b112-024242daea04 2025-05-22 01:41:10,240 - DEBUG - VMI b20281a8-36ad-11f0-b112-024242daea04 is removed from agent 10.0.0.57 2025-05-22 01:41:10,240 - INFO - Verified that pod ctest-nginx-pod-52820126 is removed in agent 2025-05-22 01:41:10,240 - INFO - Deleting pod default:ctest-nginx-pod-70236957 2025-05-22 01:41:10,256 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=8669643c-f0f8-4dc4-9671-3fdd45e17a5c 2025-05-22 01:41:10,266 - WARNING - Pod uuid 8669643c-f0f8-4dc4-9671-3fdd45e17a5c is still seen in agent 10.0.0.57 VM list 2025-05-22 01:41:15,267 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=8669643c-f0f8-4dc4-9671-3fdd45e17a5c 2025-05-22 01:41:15,274 - DEBUG - Pod 8669643c-f0f8-4dc4-9671-3fdd45e17a5c is not in agent 10.0.0.57 VM list 2025-05-22 01:41:15,274 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b145b096-36ad-11f0-b112-024242daea04 2025-05-22 01:41:15,281 - DEBUG - VMI b145b096-36ad-11f0-b112-024242daea04 is removed from agent 10.0.0.57 2025-05-22 01:41:15,282 - INFO - Verified that pod ctest-nginx-pod-70236957 is removed in agent 2025-05-22 01:41:15,282 - INFO - Deleting service : ctest-nginx-svc-44623582 2025-05-22 01:41:15,795 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_AgentXmppConnectionStatusReq? 2025-05-22 01:41:15,803 - DEBUG - No XMPP flaps were noticed during the test