2025-08-28 03:24:42,754 - INFO - Domain Default found not creating 2025-08-28 03:24:42,905 - INFO - Project ctest-TestRoutes-27899685 not found, creating it 2025-08-28 03:24:43,454 - INFO - Created Project:ctest-TestRoutes-27899685, ID : a1cea25b-2872-42cf-aa7c-73834662f493 2025-08-28 03:24:44,918 - INFO - ================================================================================ 2025-08-28 03:24:44,918 - INFO - STARTING TEST : test_rt_table_scenario_1 2025-08-28 03:24:44,918 - INFO - TEST DESCRIPTION : In a VN, launch a VM each on two computes Add a shorter prefix intf static route to VM2 Delete the VMs. The route table on compute1 should be removed 2025-08-28 03:24:46,185 - DEBUG - Nothing to compare xmpp stats {'10.0.0.65': {'10.20.0.17': '0', '10.20.0.254': '0'}, '10.0.0.27': {'10.20.0.17': '0', '10.20.0.25': '0'}} with 2025-08-28 03:24:46,185 - INFO - Initial checks done. Running the testcase now 2025-08-28 03:24:46,186 - INFO - 2025-08-28 03:24:46,888 - DEBUG - Response for create_network : {'network': {'id': '455ac98e-6a53-4088-bd2e-1691f16a0ae5', 'name': 'ctest-random-20457151', 'tenant_id': 'a1cea25b287242cfaa7c73834662f493', 'project_id': 'a1cea25b287242cfaa7c73834662f493', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestRoutes-27899685', 'ctest-random-20457151'], 'port_security_enabled': True, 'description': ''}} 2025-08-28 03:24:47,117 - DEBUG - Response for create_subnet : {'subnet': {'id': '77ca212f-57ba-4797-bf3b-d8ec0309927a', 'name': '', 'tenant_id': 'a1cea25b287242cfaa7c73834662f493', 'network_id': '455ac98e-6a53-4088-bd2e-1691f16a0ae5', 'ip_version': 4, 'cidr': '22.99.75.128/26', 'allocation_pools': [{'start': '22.99.75.130', 'end': '22.99.75.190'}], 'gateway_ip': '22.99.75.129', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '22.99.75.130', 'tags': [], 'project_id': 'a1cea25b287242cfaa7c73834662f493'}} 2025-08-28 03:24:47,143 - INFO - Created VN ctest-random-20457151 2025-08-28 03:24:47,156 - DEBUG - VN ctest-random-20457151 UUID is 455ac98e-6a53-4088-bd2e-1691f16a0ae5 2025-08-28 03:24:47,234 - DEBUG - Read virtual machine interface {'fq_name': ['default-domain', 'ctest-TestRoutes-27899685', 'ctest-vmi-56187169']} 2025-08-28 03:24:47,521 - DEBUG - Response for create_port : {'port': {'name': 'ctest-vmi-56187169', 'id': 'e3ffec50-c4eb-45ff-b45c-db87c84d0d90', 'tenant_id': 'a1cea25b287242cfaa7c73834662f493', 'network_id': '455ac98e-6a53-4088-bd2e-1691f16a0ae5', 'mac_address': '02:e3:ff:ec:50:c4', 'binding:vnic_type': 'normal', 'binding:vif_details': {'port_filter': True}, 'binding:vif_type': 'unbound', 'binding:host_id': None, 'allowed_address_pairs': [], 'fixed_ips': [{'ip_address': '22.99.75.131', 'subnet_id': '77ca212f-57ba-4797-bf3b-d8ec0309927a'}], 'security_groups': ['860752cc-71c5-411d-863f-ecbcf73f596b'], 'admin_state_up': True, 'device_id': '', 'device_owner': '', 'status': 'DOWN', 'port_security_enabled': True, 'description': '', 'project_id': 'a1cea25b287242cfaa7c73834662f493'}} 2025-08-28 03:24:47,521 - DEBUG - Created port e3ffec50-c4eb-45ff-b45c-db87c84d0d90 2025-08-28 03:24:47,614 - DEBUG - Services list from nova: [, , , ] 2025-08-28 03:24:47,614 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3956-1', 'an-jenkins-deploy-platform-ansible-os-3956-2']} 2025-08-28 03:24:47,614 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3956-1', 'an-jenkins-deploy-platform-ansible-os-3956-2']} 2025-08-28 03:24:48,766 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3956-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3956-1) 2025-08-28 03:24:49,301 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3956-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3956-2) 2025-08-28 03:24:49,541 - INFO - Created InterfaceRouteTable ['default-domain', 'ctest-TestRoutes-27899685', 'ctest-irtb-76836687'](UUID 7a7fd9af-38ae-4acc-8ba8-eb310e2a8ae5), prefixes : ['0.0.0.0/0'] 2025-08-28 03:24:49,541 - DEBUG - Sleeping for 1 seconds.. 2025-08-28 03:24:50,613 - INFO - Added intf route table 7a7fd9af-38ae-4acc-8ba8-eb310e2a8ae5 to port e3ffec50-c4eb-45ff-b45c-db87c84d0d90 2025-08-28 03:24:50,692 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-08-28 03:24:55,774 - DEBUG - VM is in ACTIVE state now 2025-08-28 03:24:55,774 - INFO - VM name : ctest-TestRoutes-27899685-13123747 2025-08-28 03:24:55,848 - DEBUG - VM ctest-TestRoutes-27899685-13123747 ID is 3ca6504a-2430-4342-b72d-c26835c89f10 2025-08-28 03:24:55,928 - DEBUG - VM is in ACTIVE state now 2025-08-28 03:24:55,961 - DEBUG - VM ctest-TestRoutes-27899685-13123747 launched on Node an-jenkins-deploy-platform-ansible-os-3956-1 2025-08-28 03:24:56,034 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10 2025-08-28 03:24:56,330 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10 2025-08-28 03:24:56,365 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10 2025-08-28 03:24:56,371 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/47dc495e-0fb0-46c3-89ce-043843544d31 2025-08-28 03:24:56,381 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10 2025-08-28 03:24:56,388 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/47dc495e-0fb0-46c3-89ce-043843544d31 2025-08-28 03:24:56,397 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/a7138bc1-9052-4e83-b022-375e19b7d7c8 2025-08-28 03:24:56,406 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 03:24:56,406 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10 2025-08-28 03:24:56,414 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/47dc495e-0fb0-46c3-89ce-043843544d31 2025-08-28 03:24:56,422 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/a7138bc1-9052-4e83-b022-375e19b7d7c8 2025-08-28 03:24:56,430 - INFO - VM ctest-TestRoutes-27899685-13123747 verfication in all API Servers passed 2025-08-28 03:24:56,457 - DEBUG - VM ctest-TestRoutes-27899685-13123747 Tap interface: {'index': '3', 'name': 'tap47dc495e-0f', 'uuid': '47dc495e-0fb0-46c3-89ce-043843544d31', 'vrf_name': 'default-domain:ctest-TestRoutes-27899685:ctest-random-20457151:ctest-random-20457151', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestRoutes-27899685:ctest-random-20457151', 'vm_uuid': '3ca6504a-2430-4342-b72d-c26835c89f10', 'vm_name': 'ctest-TestRoutes-27899685-13123747', 'ip_addr': '22.99.75.132', 'mac_addr': '02:47:dc:49:5e:0f', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '113', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestRoutes-27899685:47dc495e-0fb0-46c3-89ce-043843544d31', 'sg_uuid_list': ['860752cc-71c5-411d-863f-ecbcf73f596b'], 'static_route_list': None, 'vm_project_uuid': 'a1cea25b-2872-42cf-aa7c-73834662f493', 'admin_state': 'Enabled', 'flow_key_idx': '29', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-28 03:24:56,457 - DEBUG - Agent 10.0.0.65 vrf name: default-domain:ctest-TestRoutes-27899685:ctest-random-20457151:ctest-random-20457151 2025-08-28 03:24:56,462 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestRoutes-27899685:ctest-random-20457151:ctest-random-20457151', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestRoutes-27899685:ctest-random-20457151', 'table_label': '-1', 'vxlan_id': '12', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.193:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-08-28 03:24:56,544 - DEBUG - VM ctest-TestRoutes-27899685-13123747 : Tap interface tap47dc495e-0f is set to Active 2025-08-28 03:24:56,545 - DEBUG - Tap interface tap47dc495e-0f detail : {'index': '3', 'name': 'tap47dc495e-0f', 'uuid': '47dc495e-0fb0-46c3-89ce-043843544d31', 'vrf_name': 'default-domain:ctest-TestRoutes-27899685:ctest-random-20457151:ctest-random-20457151', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestRoutes-27899685:ctest-random-20457151', 'vm_uuid': '3ca6504a-2430-4342-b72d-c26835c89f10', 'vm_name': 'ctest-TestRoutes-27899685-13123747', 'ip_addr': '22.99.75.132', 'mac_addr': '02:47:dc:49:5e:0f', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '113', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestRoutes-27899685:47dc495e-0fb0-46c3-89ce-043843544d31', 'sg_uuid_list': ['860752cc-71c5-411d-863f-ecbcf73f596b'], 'static_route_list': None, 'vm_project_uuid': 'a1cea25b-2872-42cf-aa7c-73834662f493', 'admin_state': 'Enabled', 'flow_key_idx': '29', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-28 03:24:56,653 - DEBUG - Starting Layer 2 verification in Agent 2025-08-28 03:24:56,664 - DEBUG - Layer 2 path is seen for VM MAC 02:47:dc:49:5e:0f in agent 10.0.0.65 2025-08-28 03:24:56,665 - DEBUG - Active layer 2 route in agent is present for VMI tap47dc495e-0f 2025-08-28 03:24:56,665 - DEBUG - L2 label(29) matches bw route table and interface table 2025-08-28 03:24:56,670 - DEBUG - VN default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 is found in Agent of node 10.0.0.65 2025-08-28 03:24:56,804 - DEBUG - VRF IDs of VN default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 is consistent in agent 10.0.0.65 2025-08-28 03:24:56,804 - DEBUG - Route for VM IP ['22.99.75.132'] is consistent in agent 10.0.0.65 2025-08-28 03:24:56,804 - DEBUG - VN default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 verification for VM ctest-TestRoutes-27899685-13123747 in Agent 10.0.0.65 passed 2025-08-28 03:24:56,914 - DEBUG - Starting all layer 2 verification in agent 10.0.0.65 2025-08-28 03:24:56,926 - DEBUG - Route for VM MAC 02:47:dc:49:5e:0f is consistent in agent 10.0.0.65 2025-08-28 03:24:56,931 - DEBUG - VN default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 is found in Agent of node 10.0.0.27 2025-08-28 03:24:57,060 - DEBUG - VRF IDs of VN default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 is consistent in agent 10.0.0.27 2025-08-28 03:24:57,060 - DEBUG - Route for VM IP ['22.99.75.132'] is consistent in agent 10.0.0.27 2025-08-28 03:24:57,060 - DEBUG - VN default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 verification for VM ctest-TestRoutes-27899685-13123747 in Agent 10.0.0.27 passed 2025-08-28 03:24:57,170 - DEBUG - Starting all layer 2 verification in agent 10.0.0.27 2025-08-28 03:24:57,180 - DEBUG - Route for VM MAC 02:47:dc:49:5e:0f is consistent in agent 10.0.0.27 2025-08-28 03:25:00,367 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-08-28 03:25:00,367 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestRoutes-27899685-13123747 failed! 2025-08-28 03:25:00,380 - DEBUG - Gateway for vn default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 is 22.99.75.129 and allocation pool is NOT set 2025-08-28 03:25:02,453 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.11 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.71 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.710/3.409/5.108/1.699 ms') 2025-08-28 03:25:02,453 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-TestRoutes-27899685-13123747 passed 2025-08-28 03:25:02,453 - INFO - VM ctest-TestRoutes-27899685-13123747 verifications in Compute nodes passed 2025-08-28 03:25:02,704 - DEBUG - Validated VM route 22.99.75.132 in vrouter of 10.0.0.65 2025-08-28 03:25:02,709 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-08-28 03:25:02,816 - DEBUG - Validated VM route 22.99.75.132 in vrouter of 10.0.0.27 2025-08-28 03:25:02,820 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.193, 25; Got 10.20.0.193,25 2025-08-28 03:25:02,820 - DEBUG - Validated VM route 22.99.75.132 in vrouter of 10.0.0.27 2025-08-28 03:25:02,820 - INFO - Validated routes of VM ctest-TestRoutes-27899685-13123747 in all vrouters 2025-08-28 03:25:03,312 - DEBUG - Starting all layer2 verification in 10.0.0.19 Control Node 2025-08-28 03:25:03,435 - DEBUG - Layer2 route found for VM MAC 02:47:dc:49:5e:0f in Control-node 10.0.0.19 2025-08-28 03:25:03,436 - DEBUG - L2 Label for VM ctest-TestRoutes-27899685-13123747 same between Control-node 10.0.0.19 and Agent, Expected: 29, Seen: 29 2025-08-28 03:25:03,436 - DEBUG - Starting all layer2 verification in 10.0.0.41 Control Node 2025-08-28 03:25:03,567 - DEBUG - Layer2 route found for VM MAC 02:47:dc:49:5e:0f in Control-node 10.0.0.41 2025-08-28 03:25:03,567 - DEBUG - L2 Label for VM ctest-TestRoutes-27899685-13123747 same between Control-node 10.0.0.41 and Agent, Expected: 29, Seen: 29 2025-08-28 03:25:03,567 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-08-28 03:25:03,691 - DEBUG - Layer2 route found for VM MAC 02:47:dc:49:5e:0f in Control-node 10.0.0.38 2025-08-28 03:25:03,691 - DEBUG - L2 Label for VM ctest-TestRoutes-27899685-13123747 same between Control-node 10.0.0.38 and Agent, Expected: 29, Seen: 29 2025-08-28 03:25:03,691 - INFO - Verification in Control-nodes for VM ctest-TestRoutes-27899685-13123747 passed 2025-08-28 03:25:03,691 - DEBUG - Verifying the vm in opserver 2025-08-28 03:25:03,691 - DEBUG - Verifying in collector 10.0.0.41 ... 2025-08-28 03:25:03,692 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10?flat 2025-08-28 03:25:03,986 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10?flat 2025-08-28 03:25:04,045 - DEBUG - Failed to get VM ctest-TestRoutes-27899685-13123747, ID 3ca6504a-2430-4342-b72d-c26835c89f10 info from Opserver 2025-08-28 03:25:06,046 - DEBUG - Verifying the vm in opserver 2025-08-28 03:25:06,046 - DEBUG - Verifying in collector 10.0.0.41 ... 2025-08-28 03:25:06,046 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10?flat 2025-08-28 03:25:06,106 - DEBUG - Failed to get VM ctest-TestRoutes-27899685-13123747, ID 3ca6504a-2430-4342-b72d-c26835c89f10 info from Opserver 2025-08-28 03:25:08,107 - DEBUG - Verifying the vm in opserver 2025-08-28 03:25:08,107 - DEBUG - Verifying in collector 10.0.0.41 ... 2025-08-28 03:25:08,107 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10?flat 2025-08-28 03:25:08,183 - DEBUG - Failed to get VM ctest-TestRoutes-27899685-13123747, ID 3ca6504a-2430-4342-b72d-c26835c89f10 info from Opserver 2025-08-28 03:25:10,184 - DEBUG - Verifying the vm in opserver 2025-08-28 03:25:10,184 - DEBUG - Verifying in collector 10.0.0.41 ... 2025-08-28 03:25:10,184 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10?flat 2025-08-28 03:25:10,200 - DEBUG - Failed to get VM ctest-TestRoutes-27899685-13123747, ID 3ca6504a-2430-4342-b72d-c26835c89f10 info from Opserver 2025-08-28 03:25:12,201 - DEBUG - Verifying the vm in opserver 2025-08-28 03:25:12,201 - DEBUG - Verifying in collector 10.0.0.41 ... 2025-08-28 03:25:12,201 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10?flat 2025-08-28 03:25:12,214 - DEBUG - Failed to get VM ctest-TestRoutes-27899685-13123747, ID 3ca6504a-2430-4342-b72d-c26835c89f10 info from Opserver 2025-08-28 03:25:14,214 - DEBUG - Verifying the vm in opserver 2025-08-28 03:25:14,214 - DEBUG - Verifying in collector 10.0.0.41 ... 2025-08-28 03:25:14,214 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10?flat 2025-08-28 03:25:14,229 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestRoutes-27899685:47dc495e-0fb0-46c3-89ce-043843544d31?flat 2025-08-28 03:25:14,522 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestRoutes-27899685:47dc495e-0fb0-46c3-89ce-043843544d31?flat 2025-08-28 03:25:14,584 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestRoutes-27899685:47dc495e-0fb0-46c3-89ce-043843544d31?flat 2025-08-28 03:25:14,665 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10?flat 2025-08-28 03:25:14,683 - DEBUG - Verifying vm in vn uve 2025-08-28 03:25:14,684 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestRoutes-27899685:47dc495e-0fb0-46c3-89ce-043843544d31?flat 2025-08-28 03:25:14,747 - DEBUG - VM uve shows interface as {'fixed_ip4_list': ['22.99.75.132'], 'active': True, 'uuid': '47dc495e-0fb0-46c3-89ce-043843544d31', 'is_health_check_active': True, 'port_mirror_enabled': False, 'l2_active': True, 'tx_vlan': 65535, 'ip4_active': True, 'vn_uuid': '455ac98e-6a53-4088-bd2e-1691f16a0ae5', 'label': 25, 'vm_name': 'ctest-TestRoutes-27899685-13123747', 'vm_uuid': '3ca6504a-2430-4342-b72d-c26835c89f10', 'mac_address': '02:47:dc:49:5e:0f', 'rx_vlan': 65535, 'gateway': '22.99.75.129', 'ip_address': '22.99.75.132', 'admin_state': True, 'ip6_active': False, '__T': 1756351513262497, 'vhostuser_mode': 0, 'ip6_address': '::', 'virtual_network': 'default-domain:ctest-TestRoutes-27899685:ctest-random-20457151'} 2025-08-28 03:25:14,747 - DEBUG - VM uve shows ip address as ['22.99.75.132', '::'] 2025-08-28 03:25:14,747 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 2025-08-28 03:25:14,747 - INFO - Verifying through opserver in 10.0.0.19 2025-08-28 03:25:14,747 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestRoutes-27899685:ctest-random-20457151?flat 2025-08-28 03:25:14,766 - DEBUG - expected vm list 3ca6504a-2430-4342-b72d-c26835c89f10 2025-08-28 03:25:14,766 - DEBUG - Extracted vm list ['3ca6504a-2430-4342-b72d-c26835c89f10'] 2025-08-28 03:25:14,766 - INFO - VM 3ca6504a-2430-4342-b72d-c26835c89f10 is present in default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 2025-08-28 03:25:14,766 - DEBUG - Verifying vm in vrouter uve 2025-08-28 03:25:14,766 - DEBUG - Getting info from collector 10.0.0.41.. 2025-08-28 03:25:14,766 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10?flat 2025-08-28 03:25:14,779 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-3956-1'] 2025-08-28 03:25:14,779 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 03:25:14,814 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 03:25:15,127 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 03:25:15,199 - DEBUG - VM 3ca6504a-2430-4342-b72d-c26835c89f10 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 03:25:15,199 - DEBUG - Validated that VM ctest-TestRoutes-27899685-13123747 is in Vrouter an-jenkins-deploy-platform-ansible-os-3956-1 UVE 2025-08-28 03:25:15,199 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-08-28 03:25:15,207 - DEBUG - Expected tap interface of VM uuid 3ca6504a-2430-4342-b72d-c26835c89f10 is default-domain:ctest-TestRoutes-27899685:47dc495e-0fb0-46c3-89ce-043843544d31 2025-08-28 03:25:15,207 - DEBUG - Expected VN of VM uuid 3ca6504a-2430-4342-b72d-c26835c89f10 is default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 2025-08-28 03:25:15,207 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 03:25:15,240 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 03:25:15,305 - DEBUG - VM 3ca6504a-2430-4342-b72d-c26835c89f10 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 03:25:15,305 - INFO - tap interface default-domain:ctest-TestRoutes-27899685:47dc495e-0fb0-46c3-89ce-043843544d31 of vm 3ca6504a-2430-4342-b72d-c26835c89f10 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 03:25:15,305 - DEBUG - Connected network default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 of vm 3ca6504a-2430-4342-b72d-c26835c89f10 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 03:25:15,305 - DEBUG - Verifying the 3ca6504a-2430-4342-b72d-c26835c89f10 virtual network link through opserver 10.0.0.19 2025-08-28 03:25:15,305 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-08-28 03:25:15,318 - DEBUG - VM link and name as {'name': '3ca6504a-2430-4342-b72d-c26835c89f10', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10?flat'} 2025-08-28 03:25:15,318 - DEBUG - Verifying the 3ca6504a-2430-4342-b72d-c26835c89f10 virtual network link through opserver 10.0.0.38 2025-08-28 03:25:15,318 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-28 03:25:15,369 - DEBUG - VM link and name as {'name': '3ca6504a-2430-4342-b72d-c26835c89f10', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10?flat'} 2025-08-28 03:25:15,370 - DEBUG - Verifying the 3ca6504a-2430-4342-b72d-c26835c89f10 virtual network link through opserver 10.0.0.41 2025-08-28 03:25:15,370 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines 2025-08-28 03:25:15,380 - DEBUG - VM link and name as {'name': '3ca6504a-2430-4342-b72d-c26835c89f10', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10?flat'} 2025-08-28 03:25:15,381 - INFO - VM ctest-TestRoutes-27899685-13123747 validations in Opserver passed 2025-08-28 03:25:15,381 - INFO - Waiting for VM ctest-TestRoutes-27899685-90164645 to be up.. 2025-08-28 03:25:15,457 - DEBUG - VM is in ACTIVE state now 2025-08-28 03:25:15,457 - INFO - VM name : ctest-TestRoutes-27899685-90164645 2025-08-28 03:25:15,530 - DEBUG - VM ctest-TestRoutes-27899685-90164645 ID is e66c5812-afe3-4075-8657-a7f201d6a5e3 2025-08-28 03:25:15,530 - DEBUG - VM ctest-TestRoutes-27899685-90164645 launched on Node an-jenkins-deploy-platform-ansible-os-3956-2 2025-08-28 03:25:15,612 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/e66c5812-afe3-4075-8657-a7f201d6a5e3 2025-08-28 03:25:15,625 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/e3ffec50-c4eb-45ff-b45c-db87c84d0d90 2025-08-28 03:25:16,774 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=2.50 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.388 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.388/1.444/2.500/1.056 ms') 2025-08-28 03:25:16,774 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-TestRoutes-27899685-90164645 passed 2025-08-28 03:25:16,842 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 03:25:16,842 - DEBUG - Waiting to SSH to VM ctest-TestRoutes-27899685-90164645, IP 22.99.75.131, Port 22 2025-08-28 03:25:16,997 - DEBUG - VM ctest-TestRoutes-27899685-90164645 is ready for SSH connections 2025-08-28 03:25:16,998 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 03:25:17,304 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 03:25:17,338 - DEBUG - Requesting: http://10.0.0.19:8082/domain/692c5a68-a199-4232-a07c-1cd4918de9d4 2025-08-28 03:25:17,353 - DEBUG - Requesting: http://10.0.0.19:8082/project/a1cea25b-2872-42cf-aa7c-73834662f493 2025-08-28 03:25:17,449 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/455ac98e-6a53-4088-bd2e-1691f16a0ae5 2025-08-28 03:25:17,467 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/455ac98e-6a53-4088-bd2e-1691f16a0ae5 2025-08-28 03:25:17,477 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/bc0e0fe0-76d5-403a-a1f5-8a5404e8d02e 2025-08-28 03:25:17,487 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/bc0e0fe0-76d5-403a-a1f5-8a5404e8d02e 2025-08-28 03:25:17,495 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/34145b9b-b237-4c47-881d-d677a185e7c4 2025-08-28 03:25:17,502 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 03:25:17,502 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/455ac98e-6a53-4088-bd2e-1691f16a0ae5 2025-08-28 03:25:17,513 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/bc0e0fe0-76d5-403a-a1f5-8a5404e8d02e 2025-08-28 03:25:17,532 - INFO - Verified VN network id 12 for VN 455ac98e-6a53-4088-bd2e-1691f16a0ae5 2025-08-28 03:25:17,532 - INFO - Verifications in API Server for VN ctest-random-20457151 passed 2025-08-28 03:25:17,532 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/455ac98e-6a53-4088-bd2e-1691f16a0ae5 2025-08-28 03:25:17,541 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/bc0e0fe0-76d5-403a-a1f5-8a5404e8d02e 2025-08-28 03:25:17,550 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/bc0e0fe0-76d5-403a-a1f5-8a5404e8d02e 2025-08-28 03:25:17,557 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/34145b9b-b237-4c47-881d-d677a185e7c4 2025-08-28 03:25:17,569 - DEBUG - Control-node 10.0.0.19 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestRoutes-27899685:ctest-random-20457151', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestRoutes-27899685', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4997528350027366536', 'uuid-lslong': '13631857938173397733'}, 'enable': 'true', 'created': '2025-08-28T03:24:46', 'last-modified': '2025-08-28T03:24:47', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:30.498564'} 2025-08-28 03:25:17,575 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 03:25:17,581 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestRoutes-27899685:ctest-random-20457151', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestRoutes-27899685', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4997528350027366536', 'uuid-lslong': '13631857938173397733'}, 'enable': 'true', 'created': '2025-08-28T03:24:46', 'last-modified': '2025-08-28T03:24:47', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:30.507071'} 2025-08-28 03:25:17,588 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 03:25:17,593 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestRoutes-27899685:ctest-random-20457151', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestRoutes-27899685', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4997528350027366536', 'uuid-lslong': '13631857938173397733'}, 'enable': 'true', 'created': '2025-08-28T03:24:46', 'last-modified': '2025-08-28T03:24:47', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:30.519649'} 2025-08-28 03:25:17,599 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 03:25:17,599 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-random-20457151 passed 2025-08-28 03:25:17,599 - DEBUG - ====Verifying policy data for ctest-random-20457151 in API_Server ====== 2025-08-28 03:25:17,599 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 03:25:17,605 - DEBUG - Requesting: http://10.0.0.19:8082/domain/692c5a68-a199-4232-a07c-1cd4918de9d4 2025-08-28 03:25:17,616 - DEBUG - Requesting: http://10.0.0.19:8082/project/a1cea25b-2872-42cf-aa7c-73834662f493 2025-08-28 03:25:17,707 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/455ac98e-6a53-4088-bd2e-1691f16a0ae5 2025-08-28 03:25:17,718 - DEBUG - =>VN ctest-random-20457151 has no policy to be verified 2025-08-28 03:25:17,718 - DEBUG - Verifying the vn in opserver 2025-08-28 03:25:17,718 - DEBUG - Verifying the default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 virtual network link through opserver 10.0.0.19 2025-08-28 03:25:17,718 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2025-08-28 03:25:17,739 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestRoutes-27899685:ctest-random-20457151', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestRoutes-27899685:ctest-random-20457151?flat'} 2025-08-28 03:25:17,739 - DEBUG - Verifying the default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 virtual network link through opserver 10.0.0.38 2025-08-28 03:25:17,739 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-28 03:25:17,756 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestRoutes-27899685:ctest-random-20457151', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestRoutes-27899685:ctest-random-20457151?flat'} 2025-08-28 03:25:17,756 - DEBUG - Verifying the default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 virtual network link through opserver 10.0.0.41 2025-08-28 03:25:17,756 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-08-28 03:25:17,768 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestRoutes-27899685:ctest-random-20457151', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestRoutes-27899685:ctest-random-20457151?flat'} 2025-08-28 03:25:17,768 - INFO - Validated that VN default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 is found in opserver 2025-08-28 03:25:17,779 - DEBUG - VRF ids for VN ctest-random-20457151: {'10.0.0.65': '2', '10.0.0.27': '2'} 2025-08-28 03:25:17,779 - INFO - Deleting VM ctest-TestRoutes-27899685-13123747 2025-08-28 03:25:17,859 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 03:25:17,859 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10 2025-08-28 03:25:17,867 - DEBUG - VM ID 3ca6504a-2430-4342-b72d-c26835c89f10 of VM ctest-TestRoutes-27899685-13123747 is still found in API Server 2025-08-28 03:25:19,868 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 03:25:19,868 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10 2025-08-28 03:25:19,874 - DEBUG - Response Code: 404 2025-08-28 03:25:19,874 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10 2025-08-28 03:25:19,883 - DEBUG - Response Code: 404 2025-08-28 03:25:19,883 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/3ca6504a-2430-4342-b72d-c26835c89f10 2025-08-28 03:25:19,889 - DEBUG - Response Code: 404 2025-08-28 03:25:19,889 - INFO - VM ctest-TestRoutes-27899685-13123747 is fully removed in API-Server 2025-08-28 03:25:20,049 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-08-28 03:25:20,083 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 2025-08-28 03:25:20,194 - DEBUG - VM route 22.99.75.132 has been marked for discard in VN default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 of compute 10.0.0.27 2025-08-28 03:25:20,199 - DEBUG - Validated that vrouter 10.0.0.27 does not have VMs route for VN default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 2025-08-28 03:25:20,199 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 2025-08-28 03:25:20,199 - INFO - VM ctest-TestRoutes-27899685-13123747 is removed in Compute, and routes are removed in all compute nodes 2025-08-28 03:25:20,445 - INFO - Routes for VM ctest-TestRoutes-27899685-13123747 is removed in all control-nodes 2025-08-28 03:25:20,445 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 03:25:20,482 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 03:25:20,514 - DEBUG - VM 3ca6504a-2430-4342-b72d-c26835c89f10 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 03:25:20,514 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 03:25:20,515 - DEBUG - interface for vm 3ca6504a-2430-4342-b72d-c26835c89f10 deleted from vrouter uve ... 2025-08-28 03:25:20,515 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 03:25:23,515 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 03:25:23,547 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 03:25:23,575 - DEBUG - VM 3ca6504a-2430-4342-b72d-c26835c89f10 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 03:25:23,575 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 03:25:23,575 - DEBUG - interface for vm 3ca6504a-2430-4342-b72d-c26835c89f10 deleted from vrouter uve ... 2025-08-28 03:25:23,575 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 03:25:26,576 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 03:25:26,613 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 03:25:26,643 - DEBUG - VM 3ca6504a-2430-4342-b72d-c26835c89f10 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 03:25:26,643 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 03:25:26,643 - DEBUG - interface for vm 3ca6504a-2430-4342-b72d-c26835c89f10 deleted from vrouter uve ... 2025-08-28 03:25:26,643 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 03:25:29,644 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 03:25:29,677 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 03:25:29,710 - DEBUG - VM 3ca6504a-2430-4342-b72d-c26835c89f10 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 03:25:29,710 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 03:25:29,710 - DEBUG - interface for vm 3ca6504a-2430-4342-b72d-c26835c89f10 deleted from vrouter uve ... 2025-08-28 03:25:29,710 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 03:25:32,711 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 03:25:32,753 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 03:25:32,783 - DEBUG - VM 3ca6504a-2430-4342-b72d-c26835c89f10 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 03:25:32,783 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 03:25:32,783 - DEBUG - interface for vm 3ca6504a-2430-4342-b72d-c26835c89f10 deleted from vrouter uve ... 2025-08-28 03:25:32,783 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 03:25:35,784 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 03:25:35,818 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 03:25:35,852 - DEBUG - VM 3ca6504a-2430-4342-b72d-c26835c89f10 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 03:25:35,852 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 03:25:35,852 - DEBUG - interface for vm 3ca6504a-2430-4342-b72d-c26835c89f10 deleted from vrouter uve ... 2025-08-28 03:25:35,852 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 03:25:38,853 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 03:25:38,892 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 03:25:38,927 - DEBUG - VM 3ca6504a-2430-4342-b72d-c26835c89f10 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 03:25:38,927 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 03:25:38,927 - DEBUG - interface for vm 3ca6504a-2430-4342-b72d-c26835c89f10 deleted from vrouter uve ... 2025-08-28 03:25:38,927 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 03:25:41,928 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 03:25:41,963 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 03:25:41,992 - DEBUG - VM 3ca6504a-2430-4342-b72d-c26835c89f10 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 03:25:41,992 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 03:25:41,992 - DEBUG - interface for vm 3ca6504a-2430-4342-b72d-c26835c89f10 deleted from vrouter uve ... 2025-08-28 03:25:41,992 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 03:25:44,993 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 03:25:45,038 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 03:25:45,070 - DEBUG - VM 3ca6504a-2430-4342-b72d-c26835c89f10 is not present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 03:25:45,070 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 03:25:45,070 - DEBUG - interface for vm 3ca6504a-2430-4342-b72d-c26835c89f10 deleted from vrouter uve ... 2025-08-28 03:25:45,071 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 03:25:45,071 - DEBUG - Validated that VM 3ca6504a-2430-4342-b72d-c26835c89f10 is removed in Vrouter UVE 2025-08-28 03:25:45,071 - DEBUG - Verifying through opserver in 10.0.0.19 2025-08-28 03:25:45,071 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestRoutes-27899685:ctest-random-20457151?flat 2025-08-28 03:25:45,092 - DEBUG - 3ca6504a-2430-4342-b72d-c26835c89f10 BM not in default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 uve 2025-08-28 03:25:45,093 - DEBUG - Validated that VM 3ca6504a-2430-4342-b72d-c26835c89f10 is not present in the VN UVE 2025-08-28 03:25:45,093 - DEBUG - Verifying the 3ca6504a-2430-4342-b72d-c26835c89f10 virtual network link through opserver 10.0.0.19 2025-08-28 03:25:45,093 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-08-28 03:25:45,104 - INFO - 3ca6504a-2430-4342-b72d-c26835c89f10 vm uve deleted from opserver 2025-08-28 03:25:45,104 - DEBUG - Verifying the 3ca6504a-2430-4342-b72d-c26835c89f10 virtual network link through opserver 10.0.0.38 2025-08-28 03:25:45,104 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-28 03:25:45,117 - INFO - 3ca6504a-2430-4342-b72d-c26835c89f10 vm uve deleted from opserver 2025-08-28 03:25:45,117 - DEBUG - Verifying the 3ca6504a-2430-4342-b72d-c26835c89f10 virtual network link through opserver 10.0.0.41 2025-08-28 03:25:45,117 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines 2025-08-28 03:25:45,129 - INFO - 3ca6504a-2430-4342-b72d-c26835c89f10 vm uve deleted from opserver 2025-08-28 03:25:45,129 - INFO - Validated that VM 3ca6504a-2430-4342-b72d-c26835c89f10 is removed from Opserver 2025-08-28 03:25:45,129 - INFO - Validate that Vrouter Route table does get cleared 2025-08-28 03:25:45,201 - INFO - Removed intf route table 7a7fd9af-38ae-4acc-8ba8-eb310e2a8ae5 from port e3ffec50-c4eb-45ff-b45c-db87c84d0d90 2025-08-28 03:25:45,237 - INFO - Deleted Interface route table 7a7fd9af-38ae-4acc-8ba8-eb310e2a8ae5 2025-08-28 03:25:45,237 - INFO - Detaching port e3ffec50-c4eb-45ff-b45c-db87c84d0d90 from VM ctest-TestRoutes-27899685-90164645 2025-08-28 03:25:45,271 - INFO - Deleting VM ctest-TestRoutes-27899685-90164645 2025-08-28 03:25:45,549 - DEBUG - Response for delete_port : () 2025-08-28 03:25:45,549 - INFO - Deleted port e3ffec50-c4eb-45ff-b45c-db87c84d0d90 2025-08-28 03:25:45,549 - INFO - Deleting VN ctest-random-20457151 2025-08-28 03:25:45,710 - DEBUG - Response for deleting network () 2025-08-28 03:25:45,711 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/bc0e0fe0-76d5-403a-a1f5-8a5404e8d02e 2025-08-28 03:25:45,720 - DEBUG - Response Code: 404 2025-08-28 03:25:45,721 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 03:25:45,726 - DEBUG - Requesting: http://10.0.0.19:8082/domain/692c5a68-a199-4232-a07c-1cd4918de9d4 2025-08-28 03:25:45,780 - DEBUG - Requesting: http://10.0.0.19:8082/project/a1cea25b-2872-42cf-aa7c-73834662f493 2025-08-28 03:25:45,869 - INFO - Validated that VN ctest-random-20457151 is not found in API Server 2025-08-28 03:25:45,879 - DEBUG - VN ctest-random-20457151 is not present in Agent 10.0.0.65 2025-08-28 03:25:45,887 - DEBUG - VN ctest-random-20457151 is not present in Agent 10.0.0.27 2025-08-28 03:25:45,887 - INFO - Validated that VN ctest-random-20457151 is not in any agent 2025-08-28 03:25:45,892 - DEBUG - VRF 2 is not seen in agent 10.0.0.65 2025-08-28 03:25:45,897 - DEBUG - Vrouter 10.0.0.65 does not have vrf 2 for VN ctest-random-20457151 2025-08-28 03:25:45,900 - DEBUG - VRF 2 is not seen in agent 10.0.0.27 2025-08-28 03:25:45,904 - DEBUG - Vrouter 10.0.0.27 does not have vrf 2 for VN ctest-random-20457151 2025-08-28 03:25:45,904 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestRoutes-27899685:ctest-random-20457151 2025-08-28 03:25:45,923 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-random-20457151 info 2025-08-28 03:25:47,175 - DEBUG - No XMPP flaps were noticed during the test 2025-08-28 03:25:47,175 - INFO - END TEST : test_rt_table_scenario_1 : PASSED[0:01:03] 2025-08-28 03:25:47,175 - INFO - -------------------------------------------------------------------------------- 2025-08-28 03:25:47,875 - INFO - Deleted project: ctest-TestRoutes-27899685, ID : a1cea25b-2872-42cf-aa7c-73834662f493