2025-12-04 04:22:58,311 - INFO - Domain Default found not creating 2025-12-04 04:22:58,500 - INFO - Project ctest-TestEvpnCasesRestart-06924132 not found, creating it 2025-12-04 04:22:59,265 - INFO - Created Project:ctest-TestEvpnCasesRestart-06924132, ID : e9cf5f11-adc9-4409-ae43-9e302251b778 2025-12-04 04:23:01,470 - INFO - ================================================================================ 2025-12-04 04:23:01,470 - INFO - STARTING TEST : test_with_vxlan_encap_agent_restart 2025-12-04 04:23:01,470 - INFO - TEST DESCRIPTION : Description:Test agent restart with VXLAN Encap 1. Configure VXLAN as highest priority 2. Configure 2 VM under a VN configured with l2-l3 mode 3. Send L2 traffic and verify communication between vms 4. Restart the contrail-grouter service. 5. Send L2 Traffic and verify communication between vms Pass criteria: Step 3 and 5 should pass Maintainer: chhandak@juniper.net 2025-12-04 04:23:02,757 - DEBUG - Skipping xmpp flap check 2025-12-04 04:23:02,757 - INFO - Initial checks done. Running the testcase now 2025-12-04 04:23:02,757 - INFO - 2025-12-04 04:23:02,757 - INFO - Setting new Encap before continuing 2025-12-04 04:23:02,758 - INFO - Read the existing encap priority 2025-12-04 04:23:03,313 - INFO - Created.UUID is {"global-vrouter-config": {"uuid": "b84c431a-d13e-4ee2-903f-c691730a0ac9", "href": "http://10.0.0.23:8082/global-vrouter-config/b84c431a-d13e-4ee2-903f-c691730a0ac9"}}. VXLAN is the highest priority encap 2025-12-04 04:23:03,341 - DEBUG - Services list from nova: [, , , ] 2025-12-04 04:23:03,341 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-4840-1', 'an-jenkins-deploy-platform-ansible-os-4840-2']} 2025-12-04 04:23:03,730 - DEBUG - Response for create_network : {'network': {'id': 'db97f53b-da08-44b2-bdb7-5bb8c55fbb74', 'name': 'EVPN-VN1', 'tenant_id': 'e9cf5f11adc94409ae439e302251b778', 'project_id': 'e9cf5f11adc94409ae439e302251b778', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestEvpnCasesRestart-06924132', 'EVPN-VN1'], 'port_security_enabled': True, 'description': ''}} 2025-12-04 04:23:04,049 - DEBUG - Response for create_subnet : {'subnet': {'id': 'be7c5401-d0cd-4d6b-9d8d-d38bb75e6a4d', 'name': '', 'tenant_id': 'e9cf5f11adc94409ae439e302251b778', 'network_id': 'db97f53b-da08-44b2-bdb7-5bb8c55fbb74', 'ip_version': 4, 'cidr': '11.1.1.0/24', 'allocation_pools': [{'start': '11.1.1.2', 'end': '11.1.1.254'}], 'gateway_ip': '11.1.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '11.1.1.2', 'tags': [], 'project_id': 'e9cf5f11adc94409ae439e302251b778'}} 2025-12-04 04:23:04,078 - INFO - Created VN EVPN-VN1 2025-12-04 04:23:04,091 - DEBUG - VN EVPN-VN1 UUID is db97f53b-da08-44b2-bdb7-5bb8c55fbb74 2025-12-04 04:23:05,402 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4840-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4840-1) 2025-12-04 04:23:06,892 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4840-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4840-2) 2025-12-04 04:23:06,892 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-04 04:23:07,225 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-04 04:23:07,274 - DEBUG - Requesting: http://10.0.0.27:8082/domain/77f3106b-db9e-49b5-893e-09cac574b890 2025-12-04 04:23:07,297 - DEBUG - Requesting: http://10.0.0.27:8082/project/e9cf5f11-adc9-4409-ae43-9e302251b778 2025-12-04 04:23:07,365 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/db97f53b-da08-44b2-bdb7-5bb8c55fbb74 2025-12-04 04:23:07,379 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/db97f53b-da08-44b2-bdb7-5bb8c55fbb74 2025-12-04 04:23:07,392 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/731d0135-959f-41f8-b28d-8a308053cec2 2025-12-04 04:23:07,420 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/731d0135-959f-41f8-b28d-8a308053cec2 2025-12-04 04:23:07,438 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/bc9e07c0-c126-48c9-a1fa-fcd7e8a174e4 2025-12-04 04:23:07,448 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-04 04:23:07,449 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/db97f53b-da08-44b2-bdb7-5bb8c55fbb74 2025-12-04 04:23:07,459 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/731d0135-959f-41f8-b28d-8a308053cec2 2025-12-04 04:23:07,489 - INFO - Verified VN network id 13 for VN db97f53b-da08-44b2-bdb7-5bb8c55fbb74 2025-12-04 04:23:07,489 - INFO - Verifications in API Server for VN EVPN-VN1 passed 2025-12-04 04:23:07,489 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/db97f53b-da08-44b2-bdb7-5bb8c55fbb74 2025-12-04 04:23:07,501 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/731d0135-959f-41f8-b28d-8a308053cec2 2025-12-04 04:23:07,510 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/731d0135-959f-41f8-b28d-8a308053cec2 2025-12-04 04:23:07,520 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/bc9e07c0-c126-48c9-a1fa-fcd7e8a174e4 2025-12-04 04:23:07,537 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestEvpnCasesRestart-06924132', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15823385453199377586', 'uuid-lslong': '13670496043049925492'}, 'enable': 'true', 'created': '2025-12-04T04:23:03', 'last-modified': '2025-12-04T04:23:03', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:03.556159'} 2025-12-04 04:23:07,544 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-04 04:23:07,551 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestEvpnCasesRestart-06924132', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15823385453199377586', 'uuid-lslong': '13670496043049925492'}, 'enable': 'true', 'created': '2025-12-04T04:23:03', 'last-modified': '2025-12-04T04:23:03', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:03.569355'} 2025-12-04 04:23:07,572 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-04 04:23:07,585 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestEvpnCasesRestart-06924132', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15823385453199377586', 'uuid-lslong': '13670496043049925492'}, 'enable': 'true', 'created': '2025-12-04T04:23:03', 'last-modified': '2025-12-04T04:23:03', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:03.602319'} 2025-12-04 04:23:07,596 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-04 04:23:07,596 - INFO - On all control nodes, Config, RI and RT verification for VN EVPN-VN1 passed 2025-12-04 04:23:07,597 - DEBUG - ====Verifying policy data for EVPN-VN1 in API_Server ====== 2025-12-04 04:23:07,597 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-04 04:23:07,617 - DEBUG - Requesting: http://10.0.0.27:8082/domain/77f3106b-db9e-49b5-893e-09cac574b890 2025-12-04 04:23:07,652 - DEBUG - Requesting: http://10.0.0.27:8082/project/e9cf5f11-adc9-4409-ae43-9e302251b778 2025-12-04 04:23:07,816 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/db97f53b-da08-44b2-bdb7-5bb8c55fbb74 2025-12-04 04:23:07,841 - DEBUG - =>VN EVPN-VN1 has no policy to be verified 2025-12-04 04:23:07,841 - DEBUG - Verifying the vn in opserver 2025-12-04 04:23:07,841 - DEBUG - Verifying the default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 virtual network link through opserver 10.0.0.27 2025-12-04 04:23:07,841 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-12-04 04:23:08,218 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-12-04 04:23:08,341 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1?flat'} 2025-12-04 04:23:08,341 - DEBUG - Verifying the default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 virtual network link through opserver 10.0.0.33 2025-12-04 04:23:08,341 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-12-04 04:23:08,937 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-12-04 04:23:09,041 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1?flat'} 2025-12-04 04:23:09,041 - DEBUG - Verifying the default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 virtual network link through opserver 10.0.0.23 2025-12-04 04:23:09,041 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-12-04 04:23:09,635 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-12-04 04:23:09,723 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1?flat'} 2025-12-04 04:23:09,723 - INFO - Validated that VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is found in opserver 2025-12-04 04:23:09,738 - DEBUG - VRF ids for VN EVPN-VN1: {'10.0.0.22': '2', '10.0.0.130': '2'} 2025-12-04 04:23:09,837 - DEBUG - VM is in ACTIVE state now 2025-12-04 04:23:09,837 - INFO - VM name : EVPN_VN1_VM1 2025-12-04 04:23:09,922 - DEBUG - VM EVPN_VN1_VM1 ID is 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:23:10,008 - DEBUG - VM is in ACTIVE state now 2025-12-04 04:23:10,037 - DEBUG - VM EVPN_VN1_VM1 launched on Node an-jenkins-deploy-platform-ansible-os-4840-1 2025-12-04 04:23:10,130 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:23:10,451 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:23:10,496 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:23:10,504 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/2e6190b3-7589-4da4-a117-16eae9670168 2025-12-04 04:23:10,517 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:23:10,527 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/2e6190b3-7589-4da4-a117-16eae9670168 2025-12-04 04:23:10,539 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/8af4c969-03dc-4dbe-9dac-d183af2d21df 2025-12-04 04:23:10,551 - DEBUG - Verifying in api server 10.0.0.23 2025-12-04 04:23:10,551 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:23:10,561 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/2e6190b3-7589-4da4-a117-16eae9670168 2025-12-04 04:23:10,574 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/8af4c969-03dc-4dbe-9dac-d183af2d21df 2025-12-04 04:23:10,583 - INFO - VM EVPN_VN1_VM1 verfication in all API Servers passed 2025-12-04 04:23:10,610 - DEBUG - VM EVPN_VN1_VM1 Tap interface: {'index': '3', 'name': 'tap2e6190b3-75', 'uuid': '2e6190b3-7589-4da4-a117-16eae9670168', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'vm_uuid': '83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc', 'vm_name': 'EVPN_VN1_VM1', 'ip_addr': '11.1.1.3', 'mac_addr': '02:2e:61:90:b3:75', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '148', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:2e6190b3-7589-4da4-a117-16eae9670168', 'sg_uuid_list': ['3466f67f-9912-4f62-aeba-19b61030810a'], 'static_route_list': None, 'vm_project_uuid': 'e9cf5f11-adc9-4409-ae43-9e302251b778', 'admin_state': 'Enabled', 'flow_key_idx': '30', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-04 04:23:10,610 - DEBUG - Agent 10.0.0.22 vrf name: default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1 2025-12-04 04:23:10,618 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.201:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-12-04 04:23:10,708 - DEBUG - VM EVPN_VN1_VM1 : Tap interface tap2e6190b3-75 is set to Active 2025-12-04 04:23:10,709 - DEBUG - Tap interface tap2e6190b3-75 detail : {'index': '3', 'name': 'tap2e6190b3-75', 'uuid': '2e6190b3-7589-4da4-a117-16eae9670168', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'vm_uuid': '83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc', 'vm_name': 'EVPN_VN1_VM1', 'ip_addr': '11.1.1.3', 'mac_addr': '02:2e:61:90:b3:75', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '148', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:2e6190b3-7589-4da4-a117-16eae9670168', 'sg_uuid_list': ['3466f67f-9912-4f62-aeba-19b61030810a'], 'static_route_list': None, 'vm_project_uuid': 'e9cf5f11-adc9-4409-ae43-9e302251b778', 'admin_state': 'Enabled', 'flow_key_idx': '30', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-04 04:23:10,823 - DEBUG - Starting Layer 2 verification in Agent 2025-12-04 04:23:10,839 - DEBUG - Layer 2 path is seen for VM MAC 02:2e:61:90:b3:75 in agent 10.0.0.22 2025-12-04 04:23:10,839 - DEBUG - Active layer 2 route in agent is present for VMI tap2e6190b3-75 2025-12-04 04:23:10,840 - DEBUG - vxlan_id (13) matches bw route table and interface table 2025-12-04 04:23:10,849 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is found in Agent of node 10.0.0.22 2025-12-04 04:23:10,989 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is consistent in agent 10.0.0.22 2025-12-04 04:23:10,989 - DEBUG - Route for VM IP ['11.1.1.3'] is consistent in agent 10.0.0.22 2025-12-04 04:23:10,989 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 verification for VM EVPN_VN1_VM1 in Agent 10.0.0.22 passed 2025-12-04 04:23:11,106 - DEBUG - Starting all layer 2 verification in agent 10.0.0.22 2025-12-04 04:23:11,121 - DEBUG - Route for VM MAC 02:2e:61:90:b3:75 is consistent in agent 10.0.0.22 2025-12-04 04:23:11,128 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is found in Agent of node 10.0.0.130 2025-12-04 04:23:11,262 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is consistent in agent 10.0.0.130 2025-12-04 04:23:11,262 - DEBUG - Route for VM IP ['11.1.1.3'] is consistent in agent 10.0.0.130 2025-12-04 04:23:11,262 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 verification for VM EVPN_VN1_VM1 in Agent 10.0.0.130 passed 2025-12-04 04:23:11,374 - DEBUG - Starting all layer 2 verification in agent 10.0.0.130 2025-12-04 04:23:11,388 - DEBUG - Route for VM MAC 02:2e:61:90:b3:75 is consistent in agent 10.0.0.130 2025-12-04 04:23:14,598 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms') 2025-12-04 04:23:14,598 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-12-04 04:23:14,614 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-12-04 04:23:18,690 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-12-04 04:23:18,691 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-12-04 04:23:18,709 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-12-04 04:23:22,793 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2025-12-04 04:23:22,793 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-12-04 04:23:22,808 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-12-04 04:23:26,882 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms') 2025-12-04 04:23:26,882 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-12-04 04:23:26,898 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-12-04 04:23:30,983 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-12-04 04:23:30,983 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-12-04 04:23:30,999 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-12-04 04:23:35,070 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-12-04 04:23:35,071 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-12-04 04:23:35,089 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-12-04 04:23:39,168 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-12-04 04:23:39,168 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-12-04 04:23:39,184 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-12-04 04:23:43,266 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-12-04 04:23:43,266 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-12-04 04:23:43,282 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-12-04 04:23:47,359 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-12-04 04:23:47,360 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 failed! 2025-12-04 04:23:47,375 - DEBUG - Gateway for vn default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is 11.1.1.1 and allocation pool is NOT set 2025-12-04 04:23:51,457 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=7.08 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1015ms\r\nrtt min/avg/max/mdev = 7.083/7.083/7.083/0.000 ms') 2025-12-04 04:23:51,457 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 passed 2025-12-04 04:23:51,457 - INFO - VM EVPN_VN1_VM1 verifications in Compute nodes passed 2025-12-04 04:23:51,686 - DEBUG - Validated VM route 11.1.1.3 in vrouter of 10.0.0.22 2025-12-04 04:23:51,693 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-12-04 04:23:51,825 - DEBUG - Validated VM route 11.1.1.3 in vrouter of 10.0.0.130 2025-12-04 04:23:51,830 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.201, 25; Got 10.20.0.201,25 2025-12-04 04:23:51,830 - DEBUG - Validated VM route 11.1.1.3 in vrouter of 10.0.0.130 2025-12-04 04:23:51,830 - INFO - Validated routes of VM EVPN_VN1_VM1 in all vrouters 2025-12-04 04:23:52,339 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-12-04 04:23:52,470 - DEBUG - Layer2 route found for VM MAC 02:2e:61:90:b3:75 in Control-node 10.0.0.23 2025-12-04 04:23:52,470 - DEBUG - L2 Label for VM EVPN_VN1_VM1 same between Control-node 10.0.0.23 and Agent, Expected: 13, Seen: 13 2025-12-04 04:23:52,470 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-12-04 04:23:52,596 - DEBUG - Layer2 route found for VM MAC 02:2e:61:90:b3:75 in Control-node 10.0.0.27 2025-12-04 04:23:52,596 - DEBUG - L2 Label for VM EVPN_VN1_VM1 same between Control-node 10.0.0.27 and Agent, Expected: 13, Seen: 13 2025-12-04 04:23:52,596 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2025-12-04 04:23:52,729 - DEBUG - Layer2 route found for VM MAC 02:2e:61:90:b3:75 in Control-node 10.0.0.33 2025-12-04 04:23:52,729 - DEBUG - L2 Label for VM EVPN_VN1_VM1 same between Control-node 10.0.0.33 and Agent, Expected: 13, Seen: 13 2025-12-04 04:23:52,729 - INFO - Verification in Control-nodes for VM EVPN_VN1_VM1 passed 2025-12-04 04:23:52,729 - DEBUG - Verifying the vm in opserver 2025-12-04 04:23:52,729 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-12-04 04:23:52,729 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc?flat 2025-12-04 04:23:52,799 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-06924132:2e6190b3-7589-4da4-a117-16eae9670168?flat 2025-12-04 04:23:52,875 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-06924132:2e6190b3-7589-4da4-a117-16eae9670168?flat 2025-12-04 04:23:52,894 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc?flat 2025-12-04 04:23:52,978 - DEBUG - Verifying vm in vn uve 2025-12-04 04:23:52,978 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-06924132:2e6190b3-7589-4da4-a117-16eae9670168?flat 2025-12-04 04:23:53,055 - DEBUG - VM uve shows interface as {'uuid': '2e6190b3-7589-4da4-a117-16eae9670168', 'mac_address': '02:2e:61:90:b3:75', 'l2_active': True, 'vhostuser_mode': 0, 'port_mirror_enabled': False, 'ip6_address': '::', 'fixed_ip4_list': ['11.1.1.3'], 'vn_uuid': 'db97f53b-da08-44b2-bdb7-5bb8c55fbb74', 'vm_name': 'EVPN_VN1_VM1', 'tx_vlan': 65535, 'active': True, 'vm_uuid': '83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc', 'virtual_network': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'ip_address': '11.1.1.3', 'ip6_active': False, 'gateway': '11.1.1.1', '__T': 1764822202126712, 'ip4_active': True, 'rx_vlan': 65535, 'label': 25, 'is_health_check_active': True, 'admin_state': True} 2025-12-04 04:23:53,055 - DEBUG - VM uve shows ip address as ['11.1.1.3', '::'] 2025-12-04 04:23:53,055 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:23:53,055 - INFO - Verifying through opserver in 10.0.0.27 2025-12-04 04:23:53,055 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1?flat 2025-12-04 04:23:53,083 - DEBUG - expected vm list 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:23:53,083 - DEBUG - Extracted vm list ['83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc'] 2025-12-04 04:23:53,083 - INFO - VM 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is present in default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:23:53,084 - DEBUG - Verifying vm in vrouter uve 2025-12-04 04:23:53,084 - DEBUG - Getting info from collector 10.0.0.23.. 2025-12-04 04:23:53,084 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc?flat 2025-12-04 04:23:53,102 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4840-1'] 2025-12-04 04:23:53,102 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4840-1:Compute:contrail-vrouter-agent:0?flat 2025-12-04 04:23:53,163 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4840-1?flat 2025-12-04 04:23:53,212 - DEBUG - VM 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-1 uve 2025-12-04 04:23:53,212 - DEBUG - Validated that VM EVPN_VN1_VM1 is in Vrouter an-jenkins-deploy-platform-ansible-os-4840-1 UVE 2025-12-04 04:23:53,212 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-12-04 04:23:53,221 - DEBUG - Expected tap interface of VM uuid 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is default-domain:ctest-TestEvpnCasesRestart-06924132:2e6190b3-7589-4da4-a117-16eae9670168 2025-12-04 04:23:53,221 - DEBUG - Expected VN of VM uuid 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:23:53,221 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4840-1:Compute:contrail-vrouter-agent:0?flat 2025-12-04 04:23:53,260 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4840-1?flat 2025-12-04 04:23:53,295 - DEBUG - VM 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-1 uve 2025-12-04 04:23:53,295 - INFO - tap interface default-domain:ctest-TestEvpnCasesRestart-06924132:2e6190b3-7589-4da4-a117-16eae9670168 of vm 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-1 uve 2025-12-04 04:23:53,295 - DEBUG - Connected network default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 of vm 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-1 uve 2025-12-04 04:23:53,295 - DEBUG - Verifying the 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc virtual network link through opserver 10.0.0.27 2025-12-04 04:23:53,295 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-04 04:23:53,309 - DEBUG - VM link and name as {'name': '83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc?flat'} 2025-12-04 04:23:53,309 - DEBUG - Verifying the 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc virtual network link through opserver 10.0.0.33 2025-12-04 04:23:53,309 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-04 04:23:53,380 - DEBUG - VM link and name as {'name': '83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc?flat'} 2025-12-04 04:23:53,380 - DEBUG - Verifying the 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc virtual network link through opserver 10.0.0.23 2025-12-04 04:23:53,380 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-12-04 04:23:53,393 - DEBUG - VM link and name as {'name': '83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc?flat'} 2025-12-04 04:23:53,393 - INFO - VM EVPN_VN1_VM1 validations in Opserver passed 2025-12-04 04:23:53,489 - DEBUG - VM is in ACTIVE state now 2025-12-04 04:23:53,489 - INFO - VM name : EVPN_VN1_VM2 2025-12-04 04:23:53,583 - DEBUG - VM EVPN_VN1_VM2 ID is fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:23:53,675 - DEBUG - VM is in ACTIVE state now 2025-12-04 04:23:53,676 - DEBUG - VM EVPN_VN1_VM2 launched on Node an-jenkins-deploy-platform-ansible-os-4840-2 2025-12-04 04:23:53,780 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:23:53,791 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:23:53,799 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc 2025-12-04 04:23:53,813 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:23:53,820 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc 2025-12-04 04:23:53,839 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/95ae6efc-429a-47c2-9744-874d2b490978 2025-12-04 04:23:53,854 - DEBUG - Verifying in api server 10.0.0.23 2025-12-04 04:23:53,854 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:23:53,866 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc 2025-12-04 04:23:53,879 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/95ae6efc-429a-47c2-9744-874d2b490978 2025-12-04 04:23:53,892 - INFO - VM EVPN_VN1_VM2 verfication in all API Servers passed 2025-12-04 04:23:53,917 - DEBUG - VM EVPN_VN1_VM2 Tap interface: {'index': '3', 'name': 'tap7c9277d3-92', 'uuid': '7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'vm_uuid': 'fd0cf04c-ae57-485f-a5b0-b04f4aa036a6', 'vm_name': 'EVPN_VN1_VM2', 'ip_addr': '11.1.1.4', 'mac_addr': '02:7c:92:77:d3:92', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '114', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc', 'sg_uuid_list': ['3466f67f-9912-4f62-aeba-19b61030810a'], 'static_route_list': None, 'vm_project_uuid': 'e9cf5f11-adc9-4409-ae43-9e302251b778', 'admin_state': 'Enabled', 'flow_key_idx': '30', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-04 04:23:53,917 - DEBUG - Agent 10.0.0.130 vrf name: default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1 2025-12-04 04:23:53,928 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.252:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-12-04 04:23:54,030 - DEBUG - VM EVPN_VN1_VM2 : Tap interface tap7c9277d3-92 is set to Active 2025-12-04 04:23:54,030 - DEBUG - Tap interface tap7c9277d3-92 detail : {'index': '3', 'name': 'tap7c9277d3-92', 'uuid': '7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'vm_uuid': 'fd0cf04c-ae57-485f-a5b0-b04f4aa036a6', 'vm_name': 'EVPN_VN1_VM2', 'ip_addr': '11.1.1.4', 'mac_addr': '02:7c:92:77:d3:92', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '114', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc', 'sg_uuid_list': ['3466f67f-9912-4f62-aeba-19b61030810a'], 'static_route_list': None, 'vm_project_uuid': 'e9cf5f11-adc9-4409-ae43-9e302251b778', 'admin_state': 'Enabled', 'flow_key_idx': '30', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-04 04:23:54,145 - DEBUG - Starting Layer 2 verification in Agent 2025-12-04 04:23:54,160 - DEBUG - Layer 2 path is seen for VM MAC 02:7c:92:77:d3:92 in agent 10.0.0.130 2025-12-04 04:23:54,160 - DEBUG - Active layer 2 route in agent is present for VMI tap7c9277d3-92 2025-12-04 04:23:54,160 - DEBUG - vxlan_id (13) matches bw route table and interface table 2025-12-04 04:23:54,169 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is found in Agent of node 10.0.0.22 2025-12-04 04:23:54,308 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is consistent in agent 10.0.0.22 2025-12-04 04:23:54,308 - DEBUG - Route for VM IP ['11.1.1.4'] is consistent in agent 10.0.0.22 2025-12-04 04:23:54,309 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 verification for VM EVPN_VN1_VM2 in Agent 10.0.0.22 passed 2025-12-04 04:23:54,437 - DEBUG - Starting all layer 2 verification in agent 10.0.0.22 2025-12-04 04:23:54,453 - DEBUG - Route for VM MAC 02:7c:92:77:d3:92 is consistent in agent 10.0.0.22 2025-12-04 04:23:54,462 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is found in Agent of node 10.0.0.130 2025-12-04 04:23:54,605 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is consistent in agent 10.0.0.130 2025-12-04 04:23:54,605 - DEBUG - Route for VM IP ['11.1.1.4'] is consistent in agent 10.0.0.130 2025-12-04 04:23:54,605 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 verification for VM EVPN_VN1_VM2 in Agent 10.0.0.130 passed 2025-12-04 04:23:54,728 - DEBUG - Starting all layer 2 verification in agent 10.0.0.130 2025-12-04 04:23:54,744 - DEBUG - Route for VM MAC 02:7c:92:77:d3:92 is consistent in agent 10.0.0.130 2025-12-04 04:23:57,947 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=13.6 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1025ms\r\nrtt min/avg/max/mdev = 13.640/13.640/13.640/0.000 ms') 2025-12-04 04:23:57,947 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM2 passed 2025-12-04 04:23:57,947 - INFO - VM EVPN_VN1_VM2 verifications in Compute nodes passed 2025-12-04 04:23:58,154 - DEBUG - Validated VM route 11.1.1.4 in vrouter of 10.0.0.22 2025-12-04 04:23:58,159 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.252, 25; Got 10.20.0.252,25 2025-12-04 04:23:58,159 - DEBUG - Validated VM route 11.1.1.4 in vrouter of 10.0.0.22 2025-12-04 04:23:58,300 - DEBUG - Validated VM route 11.1.1.4 in vrouter of 10.0.0.130 2025-12-04 04:23:58,305 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-12-04 04:23:58,305 - INFO - Validated routes of VM EVPN_VN1_VM2 in all vrouters 2025-12-04 04:23:58,804 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-12-04 04:23:58,932 - DEBUG - Layer2 route found for VM MAC 02:7c:92:77:d3:92 in Control-node 10.0.0.23 2025-12-04 04:23:58,932 - DEBUG - L2 Label for VM EVPN_VN1_VM2 same between Control-node 10.0.0.23 and Agent, Expected: 13, Seen: 13 2025-12-04 04:23:58,932 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-12-04 04:23:59,061 - DEBUG - Layer2 route found for VM MAC 02:7c:92:77:d3:92 in Control-node 10.0.0.27 2025-12-04 04:23:59,061 - DEBUG - L2 Label for VM EVPN_VN1_VM2 same between Control-node 10.0.0.27 and Agent, Expected: 13, Seen: 13 2025-12-04 04:23:59,061 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2025-12-04 04:23:59,193 - DEBUG - Layer2 route found for VM MAC 02:7c:92:77:d3:92 in Control-node 10.0.0.33 2025-12-04 04:23:59,193 - DEBUG - L2 Label for VM EVPN_VN1_VM2 same between Control-node 10.0.0.33 and Agent, Expected: 13, Seen: 13 2025-12-04 04:23:59,193 - INFO - Verification in Control-nodes for VM EVPN_VN1_VM2 passed 2025-12-04 04:23:59,194 - DEBUG - Verifying the vm in opserver 2025-12-04 04:23:59,194 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-12-04 04:23:59,194 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat 2025-12-04 04:23:59,209 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-06924132:7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc?flat 2025-12-04 04:23:59,227 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-06924132:7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc?flat 2025-12-04 04:23:59,246 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat 2025-12-04 04:23:59,261 - DEBUG - Verifying vm in vn uve 2025-12-04 04:23:59,261 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-06924132:7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc?flat 2025-12-04 04:23:59,279 - DEBUG - VM uve shows interface as {'active': True, 'virtual_network': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'ip6_address': '::', 'fixed_ip4_list': ['11.1.1.4'], 'admin_state': True, 'port_mirror_enabled': False, 'is_health_check_active': True, 'tx_vlan': 65535, 'gateway': '11.1.1.1', 'uuid': '7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc', 'ip_address': '11.1.1.4', '__T': 1764822204006433, 'ip6_active': False, 'mac_address': '02:7c:92:77:d3:92', 'vhostuser_mode': 0, 'vm_uuid': 'fd0cf04c-ae57-485f-a5b0-b04f4aa036a6', 'l2_active': True, 'ip4_active': True, 'rx_vlan': 65535, 'vn_uuid': 'db97f53b-da08-44b2-bdb7-5bb8c55fbb74', 'label': 25, 'vm_name': 'EVPN_VN1_VM2'} 2025-12-04 04:23:59,280 - DEBUG - VM uve shows ip address as ['11.1.1.4', '::'] 2025-12-04 04:23:59,280 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:23:59,280 - INFO - Verifying through opserver in 10.0.0.27 2025-12-04 04:23:59,280 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1?flat 2025-12-04 04:23:59,307 - DEBUG - expected vm list fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:23:59,307 - DEBUG - Extracted vm list ['fd0cf04c-ae57-485f-a5b0-b04f4aa036a6'] 2025-12-04 04:23:59,307 - INFO - VM fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is present in default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:23:59,307 - DEBUG - Verifying vm in vrouter uve 2025-12-04 04:23:59,307 - DEBUG - Getting info from collector 10.0.0.23.. 2025-12-04 04:23:59,307 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat 2025-12-04 04:23:59,322 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4840-2'] 2025-12-04 04:23:59,322 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4840-2:Compute:contrail-vrouter-agent:0?flat 2025-12-04 04:23:59,355 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4840-2?flat 2025-12-04 04:23:59,384 - DEBUG - VM fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-2 uve 2025-12-04 04:23:59,384 - DEBUG - Validated that VM EVPN_VN1_VM2 is in Vrouter an-jenkins-deploy-platform-ansible-os-4840-2 UVE 2025-12-04 04:23:59,384 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-12-04 04:23:59,397 - DEBUG - Expected tap interface of VM uuid fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is default-domain:ctest-TestEvpnCasesRestart-06924132:7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc 2025-12-04 04:23:59,398 - DEBUG - Expected VN of VM uuid fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:23:59,398 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4840-2:Compute:contrail-vrouter-agent:0?flat 2025-12-04 04:23:59,431 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4840-2?flat 2025-12-04 04:23:59,459 - DEBUG - VM fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-2 uve 2025-12-04 04:23:59,459 - INFO - tap interface default-domain:ctest-TestEvpnCasesRestart-06924132:7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc of vm fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-2 uve 2025-12-04 04:23:59,459 - DEBUG - Connected network default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 of vm fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-2 uve 2025-12-04 04:23:59,459 - DEBUG - Verifying the fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 virtual network link through opserver 10.0.0.27 2025-12-04 04:23:59,459 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-04 04:23:59,472 - DEBUG - VM link and name as {'name': 'fd0cf04c-ae57-485f-a5b0-b04f4aa036a6', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat'} 2025-12-04 04:23:59,472 - DEBUG - Verifying the fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 virtual network link through opserver 10.0.0.33 2025-12-04 04:23:59,472 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-04 04:23:59,486 - DEBUG - VM link and name as {'name': 'fd0cf04c-ae57-485f-a5b0-b04f4aa036a6', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat'} 2025-12-04 04:23:59,486 - DEBUG - Verifying the fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 virtual network link through opserver 10.0.0.23 2025-12-04 04:23:59,486 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-12-04 04:23:59,499 - DEBUG - VM link and name as {'name': 'fd0cf04c-ae57-485f-a5b0-b04f4aa036a6', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat'} 2025-12-04 04:23:59,500 - INFO - VM EVPN_VN1_VM2 validations in Opserver passed 2025-12-04 04:23:59,500 - INFO - Waiting for VM EVPN_VN1_VM1 to be up.. 2025-12-04 04:23:59,500 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-04 04:23:59,500 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:23:59,508 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/2e6190b3-7589-4da4-a117-16eae9670168 2025-12-04 04:24:00,674 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=3.79 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=3.67 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 3.668/3.730/3.793/0.062 ms') 2025-12-04 04:24:00,674 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 passed 2025-12-04 04:24:00,748 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-04 04:24:00,748 - DEBUG - Waiting to SSH to VM EVPN_VN1_VM1, IP 11.1.1.3, Port 22 2025-12-04 04:24:00,815 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-04 04:24:00,914 - DEBUG - VM EVPN_VN1_VM1 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-04 04:24:05,915 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-04 04:24:05,915 - DEBUG - Waiting to SSH to VM EVPN_VN1_VM1, IP 11.1.1.3, Port 22 2025-12-04 04:24:05,982 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-04 04:24:06,078 - DEBUG - VM EVPN_VN1_VM1 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-04 04:24:11,078 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-04 04:24:11,079 - DEBUG - Waiting to SSH to VM EVPN_VN1_VM1, IP 11.1.1.3, Port 22 2025-12-04 04:24:11,146 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-04 04:24:11,264 - DEBUG - VM EVPN_VN1_VM1 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-04 04:24:16,265 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-04 04:24:16,265 - DEBUG - Waiting to SSH to VM EVPN_VN1_VM1, IP 11.1.1.3, Port 22 2025-12-04 04:24:16,438 - DEBUG - VM EVPN_VN1_VM1 is ready for SSH connections 2025-12-04 04:24:16,438 - INFO - Waiting for VM EVPN_VN1_VM2 to be up.. 2025-12-04 04:24:16,438 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-04 04:24:16,438 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:24:16,450 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc 2025-12-04 04:24:17,615 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=2.84 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.66 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.655/2.247/2.840/0.592 ms') 2025-12-04 04:24:17,615 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM2 passed 2025-12-04 04:24:17,687 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-04 04:24:17,687 - DEBUG - Waiting to SSH to VM EVPN_VN1_VM2, IP 11.1.1.4, Port 22 2025-12-04 04:24:17,754 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-04 04:24:17,845 - DEBUG - VM EVPN_VN1_VM2 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-04 04:24:22,845 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-04 04:24:22,846 - DEBUG - Waiting to SSH to VM EVPN_VN1_VM2, IP 11.1.1.4, Port 22 2025-12-04 04:24:23,023 - DEBUG - VM EVPN_VN1_VM2 is ready for SSH connections 2025-12-04 04:24:23,123 - DEBUG - Executing command: sudo tcpdump -nni tap7c9277d3-92 -U ether src 02:2e:61:90:b3:75 -w /tmp/tap7c9277d3-92_ctest-random-61436629.pcap 2025-12-04 04:24:23,567 - INFO - waiting to get tcpdump started 2025-12-04 04:24:46,575 - DEBUG - file copy failed retrying 2025-12-04 04:24:49,576 - DEBUG - file copy failed retrying 2025-12-04 04:25:31,841 - DEBUG - file transfer done ['/tmp/program.py'] 2025-12-04 04:25:31,841 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-04 04:25:31,841 - DEBUG - Copied file /tmp/tmplesg6l0s/program.py to 11.1.1.3 2025-12-04 04:25:31,841 - DEBUG - Running remote_cmd, Cmd : python /tmp/program.py, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.22, gateway password: c0ntrail123 2025-12-04 04:25:31,841 - DEBUG - python /tmp/program.py 2025-12-04 04:25:34,545 - DEBUG - WARNING: No route found for IPv6 destination :: (no default route?) .......... Sent 10 packets. 2025-12-04 04:25:34,545 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7c9277d3-92_ctest-random-61436629.pcap ether host 02:7c:92:77:d3:92 | grep -c length 2025-12-04 04:25:34,566 - DEBUG - STDOUT: 0 2025-12-04 04:25:34,566 - DEBUG - STDERR: reading from file /tmp/tap7c9277d3-92_ctest-random-61436629.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-04 04:25:34,567 - WARNING - 0 packets are found in tcpdump output file /tmp/tap7c9277d3-92_ctest-random-61436629.pcap but expected 10 2025-12-04 04:25:36,567 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7c9277d3-92_ctest-random-61436629.pcap ether host 02:7c:92:77:d3:92 | grep -c length 2025-12-04 04:25:36,585 - DEBUG - STDOUT: 0 2025-12-04 04:25:36,585 - DEBUG - STDERR: reading from file /tmp/tap7c9277d3-92_ctest-random-61436629.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-04 04:25:36,586 - WARNING - 0 packets are found in tcpdump output file /tmp/tap7c9277d3-92_ctest-random-61436629.pcap but expected 10 2025-12-04 04:25:38,587 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7c9277d3-92_ctest-random-61436629.pcap ether host 02:7c:92:77:d3:92 | grep -c length 2025-12-04 04:25:38,605 - DEBUG - STDOUT: 0 2025-12-04 04:25:38,605 - DEBUG - STDERR: reading from file /tmp/tap7c9277d3-92_ctest-random-61436629.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-04 04:25:38,605 - WARNING - 0 packets are found in tcpdump output file /tmp/tap7c9277d3-92_ctest-random-61436629.pcap but expected 10 2025-12-04 04:25:40,606 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7c9277d3-92_ctest-random-61436629.pcap ether host 02:7c:92:77:d3:92 | grep -c length 2025-12-04 04:25:40,627 - DEBUG - STDOUT: 0 2025-12-04 04:25:40,627 - DEBUG - STDERR: reading from file /tmp/tap7c9277d3-92_ctest-random-61436629.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-04 04:25:40,627 - WARNING - 0 packets are found in tcpdump output file /tmp/tap7c9277d3-92_ctest-random-61436629.pcap but expected 10 2025-12-04 04:25:42,627 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7c9277d3-92_ctest-random-61436629.pcap ether host 02:7c:92:77:d3:92 | grep -c length 2025-12-04 04:25:42,644 - DEBUG - STDOUT: 0 2025-12-04 04:25:42,644 - DEBUG - STDERR: reading from file /tmp/tap7c9277d3-92_ctest-random-61436629.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-04 04:25:42,644 - WARNING - 0 packets are found in tcpdump output file /tmp/tap7c9277d3-92_ctest-random-61436629.pcap but expected 10 2025-12-04 04:25:44,645 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7c9277d3-92_ctest-random-61436629.pcap ether host 02:7c:92:77:d3:92 | grep -c length 2025-12-04 04:25:44,662 - DEBUG - STDOUT: 0 2025-12-04 04:25:44,662 - DEBUG - STDERR: reading from file /tmp/tap7c9277d3-92_ctest-random-61436629.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-04 04:25:44,662 - WARNING - 0 packets are found in tcpdump output file /tmp/tap7c9277d3-92_ctest-random-61436629.pcap but expected 10 2025-12-04 04:25:46,663 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7c9277d3-92_ctest-random-61436629.pcap ether host 02:7c:92:77:d3:92 | grep -c length 2025-12-04 04:25:46,681 - DEBUG - STDOUT: 0 2025-12-04 04:25:46,682 - DEBUG - STDERR: reading from file /tmp/tap7c9277d3-92_ctest-random-61436629.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-04 04:25:46,682 - WARNING - 0 packets are found in tcpdump output file /tmp/tap7c9277d3-92_ctest-random-61436629.pcap but expected 10 2025-12-04 04:25:46,682 - INFO - Will restart compute services now 2025-12-04 04:25:50,070 - INFO - 10.0.0.22 2025-12-04 04:25:50,075 - INFO - agent:initializing 2025-12-04 04:25:50,075 - DEBUG - defaultdict(, {'10.0.0.22': {'agent': {'status': 'initializing', 'description': None}}}) 2025-12-04 04:25:50,075 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-04 04:25:55,188 - INFO - 10.0.0.22 2025-12-04 04:25:55,195 - INFO - agent:active 2025-12-04 04:25:55,195 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.22 2025-12-04 04:25:59,132 - INFO - 10.0.0.130 2025-12-04 04:25:59,137 - INFO - agent:initializing 2025-12-04 04:25:59,137 - DEBUG - defaultdict(, {'10.0.0.130': {'agent': {'status': 'initializing', 'description': None}}}) 2025-12-04 04:25:59,137 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-04 04:26:04,238 - INFO - 10.0.0.130 2025-12-04 04:26:04,246 - INFO - agent:active 2025-12-04 04:26:04,246 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.130 2025-12-04 04:26:14,247 - INFO - Verifying L2 route and other VM verification after restart 2025-12-04 04:26:14,341 - DEBUG - VM is in ACTIVE state now 2025-12-04 04:26:14,341 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-04 04:26:14,341 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:26:14,350 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/2e6190b3-7589-4da4-a117-16eae9670168 2025-12-04 04:26:14,362 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:26:14,372 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/2e6190b3-7589-4da4-a117-16eae9670168 2025-12-04 04:26:14,385 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/8af4c969-03dc-4dbe-9dac-d183af2d21df 2025-12-04 04:26:14,396 - DEBUG - Verifying in api server 10.0.0.23 2025-12-04 04:26:14,396 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:26:14,406 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/2e6190b3-7589-4da4-a117-16eae9670168 2025-12-04 04:26:14,417 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/8af4c969-03dc-4dbe-9dac-d183af2d21df 2025-12-04 04:26:14,429 - INFO - VM EVPN_VN1_VM1 verfication in all API Servers passed 2025-12-04 04:26:14,453 - DEBUG - VM EVPN_VN1_VM1 Tap interface: {'index': '3', 'name': 'tap2e6190b3-75', 'uuid': '2e6190b3-7589-4da4-a117-16eae9670168', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '23', 'l2_label': '27', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'vm_uuid': '83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc', 'vm_name': 'EVPN_VN1_VM1', 'ip_addr': '11.1.1.3', 'mac_addr': '02:2e:61:90:b3:75', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '148', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:2e6190b3-7589-4da4-a117-16eae9670168', 'sg_uuid_list': ['3466f67f-9912-4f62-aeba-19b61030810a'], 'static_route_list': None, 'vm_project_uuid': 'e9cf5f11-adc9-4409-ae43-9e302251b778', 'admin_state': 'Enabled', 'flow_key_idx': '22', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-04 04:26:14,453 - DEBUG - Agent 10.0.0.22 vrf name: default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1 2025-12-04 04:26:14,460 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.201:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-12-04 04:26:14,549 - DEBUG - VM EVPN_VN1_VM1 : Tap interface tap2e6190b3-75 is set to Active 2025-12-04 04:26:14,550 - DEBUG - Tap interface tap2e6190b3-75 detail : {'index': '3', 'name': 'tap2e6190b3-75', 'uuid': '2e6190b3-7589-4da4-a117-16eae9670168', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '23', 'l2_label': '27', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'vm_uuid': '83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc', 'vm_name': 'EVPN_VN1_VM1', 'ip_addr': '11.1.1.3', 'mac_addr': '02:2e:61:90:b3:75', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '148', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:2e6190b3-7589-4da4-a117-16eae9670168', 'sg_uuid_list': ['3466f67f-9912-4f62-aeba-19b61030810a'], 'static_route_list': None, 'vm_project_uuid': 'e9cf5f11-adc9-4409-ae43-9e302251b778', 'admin_state': 'Enabled', 'flow_key_idx': '22', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-04 04:26:14,667 - DEBUG - Starting Layer 2 verification in Agent 2025-12-04 04:26:14,680 - DEBUG - Layer 2 path is seen for VM MAC 02:2e:61:90:b3:75 in agent 10.0.0.22 2025-12-04 04:26:14,681 - DEBUG - Active layer 2 route in agent is present for VMI tap2e6190b3-75 2025-12-04 04:26:14,681 - DEBUG - vxlan_id (13) matches bw route table and interface table 2025-12-04 04:26:14,689 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is found in Agent of node 10.0.0.22 2025-12-04 04:26:14,822 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is consistent in agent 10.0.0.22 2025-12-04 04:26:14,822 - DEBUG - Route for VM IP ['11.1.1.3'] is consistent in agent 10.0.0.22 2025-12-04 04:26:14,822 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 verification for VM EVPN_VN1_VM1 in Agent 10.0.0.22 passed 2025-12-04 04:26:14,934 - DEBUG - Starting all layer 2 verification in agent 10.0.0.22 2025-12-04 04:26:14,946 - DEBUG - Route for VM MAC 02:2e:61:90:b3:75 is consistent in agent 10.0.0.22 2025-12-04 04:26:14,953 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is found in Agent of node 10.0.0.130 2025-12-04 04:26:15,085 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is consistent in agent 10.0.0.130 2025-12-04 04:26:15,085 - DEBUG - Route for VM IP ['11.1.1.3'] is consistent in agent 10.0.0.130 2025-12-04 04:26:15,085 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 verification for VM EVPN_VN1_VM1 in Agent 10.0.0.130 passed 2025-12-04 04:26:15,199 - DEBUG - Starting all layer 2 verification in agent 10.0.0.130 2025-12-04 04:26:15,210 - DEBUG - Route for VM MAC 02:2e:61:90:b3:75 is consistent in agent 10.0.0.130 2025-12-04 04:26:16,391 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=1.97 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.637 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.637/1.305/1.974/0.668 ms') 2025-12-04 04:26:16,391 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 passed 2025-12-04 04:26:16,391 - INFO - VM EVPN_VN1_VM1 verifications in Compute nodes passed 2025-12-04 04:26:16,587 - DEBUG - Validated VM route 11.1.1.3 in vrouter of 10.0.0.22 2025-12-04 04:26:16,593 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-12-04 04:26:16,705 - DEBUG - Validated VM route 11.1.1.3 in vrouter of 10.0.0.130 2025-12-04 04:26:16,711 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.201, 23; Got 10.20.0.201,23 2025-12-04 04:26:16,711 - DEBUG - Validated VM route 11.1.1.3 in vrouter of 10.0.0.130 2025-12-04 04:26:16,711 - INFO - Validated routes of VM EVPN_VN1_VM1 in all vrouters 2025-12-04 04:26:17,216 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-12-04 04:26:17,344 - DEBUG - Layer2 route found for VM MAC 02:2e:61:90:b3:75 in Control-node 10.0.0.23 2025-12-04 04:26:17,344 - DEBUG - L2 Label for VM EVPN_VN1_VM1 same between Control-node 10.0.0.23 and Agent, Expected: 13, Seen: 13 2025-12-04 04:26:17,344 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-12-04 04:26:17,479 - DEBUG - Layer2 route found for VM MAC 02:2e:61:90:b3:75 in Control-node 10.0.0.27 2025-12-04 04:26:17,479 - DEBUG - L2 Label for VM EVPN_VN1_VM1 same between Control-node 10.0.0.27 and Agent, Expected: 13, Seen: 13 2025-12-04 04:26:17,479 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2025-12-04 04:26:17,610 - DEBUG - Layer2 route found for VM MAC 02:2e:61:90:b3:75 in Control-node 10.0.0.33 2025-12-04 04:26:17,610 - DEBUG - L2 Label for VM EVPN_VN1_VM1 same between Control-node 10.0.0.33 and Agent, Expected: 13, Seen: 13 2025-12-04 04:26:17,610 - INFO - Verification in Control-nodes for VM EVPN_VN1_VM1 passed 2025-12-04 04:26:17,610 - DEBUG - Verifying the vm in opserver 2025-12-04 04:26:17,610 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-12-04 04:26:17,610 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc?flat 2025-12-04 04:26:17,626 - DEBUG - Failed to get VM EVPN_VN1_VM1, ID 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc info from Opserver 2025-12-04 04:26:19,627 - DEBUG - Verifying the vm in opserver 2025-12-04 04:26:19,627 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-12-04 04:26:19,627 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc?flat 2025-12-04 04:26:19,644 - DEBUG - Failed to get VM EVPN_VN1_VM1, ID 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc info from Opserver 2025-12-04 04:26:21,645 - DEBUG - Verifying the vm in opserver 2025-12-04 04:26:21,645 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-12-04 04:26:21,646 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc?flat 2025-12-04 04:26:21,663 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-06924132:2e6190b3-7589-4da4-a117-16eae9670168?flat 2025-12-04 04:26:21,684 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-06924132:2e6190b3-7589-4da4-a117-16eae9670168?flat 2025-12-04 04:26:21,705 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc?flat 2025-12-04 04:26:21,721 - DEBUG - Verifying vm in vn uve 2025-12-04 04:26:21,721 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-06924132:2e6190b3-7589-4da4-a117-16eae9670168?flat 2025-12-04 04:26:21,743 - DEBUG - VM uve shows interface as {'uuid': '2e6190b3-7589-4da4-a117-16eae9670168', 'mac_address': '02:2e:61:90:b3:75', 'l2_active': True, 'vhostuser_mode': 0, 'port_mirror_enabled': False, 'ip6_address': '::', 'fixed_ip4_list': ['11.1.1.3'], 'vn_uuid': 'db97f53b-da08-44b2-bdb7-5bb8c55fbb74', 'vm_name': 'EVPN_VN1_VM1', 'tx_vlan': 65535, 'active': True, 'vm_uuid': '83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc', 'virtual_network': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'ip_address': '11.1.1.3', 'ip6_active': False, 'gateway': '11.1.1.1', '__T': 1764822380738474, 'ip4_active': True, 'rx_vlan': 65535, 'label': 23, 'is_health_check_active': True, 'admin_state': True} 2025-12-04 04:26:21,743 - DEBUG - VM uve shows ip address as ['11.1.1.3', '::'] 2025-12-04 04:26:21,743 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:26:21,743 - INFO - Verifying through opserver in 10.0.0.27 2025-12-04 04:26:21,743 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1?flat 2025-12-04 04:26:21,770 - DEBUG - expected vm list 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:26:21,770 - DEBUG - Extracted vm list ['83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc'] 2025-12-04 04:26:21,770 - INFO - VM 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is present in default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:26:21,770 - DEBUG - Verifying vm in vrouter uve 2025-12-04 04:26:21,770 - DEBUG - Getting info from collector 10.0.0.23.. 2025-12-04 04:26:21,770 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc?flat 2025-12-04 04:26:21,787 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4840-1'] 2025-12-04 04:26:21,787 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4840-1:Compute:contrail-vrouter-agent:0?flat 2025-12-04 04:26:21,827 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4840-1?flat 2025-12-04 04:26:21,862 - DEBUG - VM 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-1 uve 2025-12-04 04:26:21,862 - DEBUG - Validated that VM EVPN_VN1_VM1 is in Vrouter an-jenkins-deploy-platform-ansible-os-4840-1 UVE 2025-12-04 04:26:21,862 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-12-04 04:26:21,870 - DEBUG - Expected tap interface of VM uuid 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is default-domain:ctest-TestEvpnCasesRestart-06924132:2e6190b3-7589-4da4-a117-16eae9670168 2025-12-04 04:26:21,870 - DEBUG - Expected VN of VM uuid 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:26:21,870 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4840-1:Compute:contrail-vrouter-agent:0?flat 2025-12-04 04:26:21,915 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4840-1?flat 2025-12-04 04:26:21,950 - DEBUG - VM 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-1 uve 2025-12-04 04:26:21,950 - INFO - tap interface default-domain:ctest-TestEvpnCasesRestart-06924132:2e6190b3-7589-4da4-a117-16eae9670168 of vm 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-1 uve 2025-12-04 04:26:21,950 - DEBUG - Connected network default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 of vm 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-1 uve 2025-12-04 04:26:21,951 - DEBUG - Verifying the 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc virtual network link through opserver 10.0.0.27 2025-12-04 04:26:21,951 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-04 04:26:21,967 - DEBUG - VM link and name as {'name': '83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc?flat'} 2025-12-04 04:26:21,967 - DEBUG - Verifying the 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc virtual network link through opserver 10.0.0.33 2025-12-04 04:26:21,967 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-04 04:26:21,983 - DEBUG - VM link and name as {'name': '83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc?flat'} 2025-12-04 04:26:21,983 - DEBUG - Verifying the 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc virtual network link through opserver 10.0.0.23 2025-12-04 04:26:21,983 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-12-04 04:26:21,996 - DEBUG - VM link and name as {'name': '83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc?flat'} 2025-12-04 04:26:21,996 - INFO - VM EVPN_VN1_VM1 validations in Opserver passed 2025-12-04 04:26:22,091 - DEBUG - VM is in ACTIVE state now 2025-12-04 04:26:22,091 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-04 04:26:22,091 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:26:22,101 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc 2025-12-04 04:26:22,111 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:26:22,120 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc 2025-12-04 04:26:22,133 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/95ae6efc-429a-47c2-9744-874d2b490978 2025-12-04 04:26:22,144 - DEBUG - Verifying in api server 10.0.0.23 2025-12-04 04:26:22,144 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:26:22,152 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc 2025-12-04 04:26:22,162 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/95ae6efc-429a-47c2-9744-874d2b490978 2025-12-04 04:26:22,171 - INFO - VM EVPN_VN1_VM2 verfication in all API Servers passed 2025-12-04 04:26:22,192 - DEBUG - VM EVPN_VN1_VM2 Tap interface: {'index': '3', 'name': 'tap7c9277d3-92', 'uuid': '7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '22', 'l2_label': '26', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'vm_uuid': 'fd0cf04c-ae57-485f-a5b0-b04f4aa036a6', 'vm_name': 'EVPN_VN1_VM2', 'ip_addr': '11.1.1.4', 'mac_addr': '02:7c:92:77:d3:92', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '114', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc', 'sg_uuid_list': ['3466f67f-9912-4f62-aeba-19b61030810a'], 'static_route_list': None, 'vm_project_uuid': 'e9cf5f11-adc9-4409-ae43-9e302251b778', 'admin_state': 'Enabled', 'flow_key_idx': '19', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-04 04:26:22,192 - DEBUG - Agent 10.0.0.130 vrf name: default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1 2025-12-04 04:26:22,201 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.252:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-12-04 04:26:22,286 - DEBUG - VM EVPN_VN1_VM2 : Tap interface tap7c9277d3-92 is set to Active 2025-12-04 04:26:22,286 - DEBUG - Tap interface tap7c9277d3-92 detail : {'index': '3', 'name': 'tap7c9277d3-92', 'uuid': '7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc', 'vrf_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1:EVPN-VN1', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '22', 'l2_label': '26', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'vm_uuid': 'fd0cf04c-ae57-485f-a5b0-b04f4aa036a6', 'vm_name': 'EVPN_VN1_VM2', 'ip_addr': '11.1.1.4', 'mac_addr': '02:7c:92:77:d3:92', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '114', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestEvpnCasesRestart-06924132:7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc', 'sg_uuid_list': ['3466f67f-9912-4f62-aeba-19b61030810a'], 'static_route_list': None, 'vm_project_uuid': 'e9cf5f11-adc9-4409-ae43-9e302251b778', 'admin_state': 'Enabled', 'flow_key_idx': '19', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-04 04:26:22,400 - DEBUG - Starting Layer 2 verification in Agent 2025-12-04 04:26:22,411 - DEBUG - Layer 2 path is seen for VM MAC 02:7c:92:77:d3:92 in agent 10.0.0.130 2025-12-04 04:26:22,411 - DEBUG - Active layer 2 route in agent is present for VMI tap7c9277d3-92 2025-12-04 04:26:22,411 - DEBUG - vxlan_id (13) matches bw route table and interface table 2025-12-04 04:26:22,417 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is found in Agent of node 10.0.0.22 2025-12-04 04:26:22,548 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is consistent in agent 10.0.0.22 2025-12-04 04:26:22,548 - DEBUG - Route for VM IP ['11.1.1.4'] is consistent in agent 10.0.0.22 2025-12-04 04:26:22,548 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 verification for VM EVPN_VN1_VM2 in Agent 10.0.0.22 passed 2025-12-04 04:26:22,667 - DEBUG - Starting all layer 2 verification in agent 10.0.0.22 2025-12-04 04:26:22,677 - DEBUG - Route for VM MAC 02:7c:92:77:d3:92 is consistent in agent 10.0.0.22 2025-12-04 04:26:22,684 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is found in Agent of node 10.0.0.130 2025-12-04 04:26:22,815 - DEBUG - VRF IDs of VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 is consistent in agent 10.0.0.130 2025-12-04 04:26:22,816 - DEBUG - Route for VM IP ['11.1.1.4'] is consistent in agent 10.0.0.130 2025-12-04 04:26:22,816 - DEBUG - VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 verification for VM EVPN_VN1_VM2 in Agent 10.0.0.130 passed 2025-12-04 04:26:22,931 - DEBUG - Starting all layer 2 verification in agent 10.0.0.130 2025-12-04 04:26:22,943 - DEBUG - Route for VM MAC 02:7c:92:77:d3:92 is consistent in agent 10.0.0.130 2025-12-04 04:26:24,116 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=2.96 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.507 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.507/1.733/2.959/1.226 ms') 2025-12-04 04:26:24,116 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM2 passed 2025-12-04 04:26:24,116 - INFO - VM EVPN_VN1_VM2 verifications in Compute nodes passed 2025-12-04 04:26:24,329 - DEBUG - Validated VM route 11.1.1.4 in vrouter of 10.0.0.22 2025-12-04 04:26:24,334 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.252, 22; Got 10.20.0.252,22 2025-12-04 04:26:24,334 - DEBUG - Validated VM route 11.1.1.4 in vrouter of 10.0.0.22 2025-12-04 04:26:24,450 - DEBUG - Validated VM route 11.1.1.4 in vrouter of 10.0.0.130 2025-12-04 04:26:24,456 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-12-04 04:26:24,456 - INFO - Validated routes of VM EVPN_VN1_VM2 in all vrouters 2025-12-04 04:26:24,942 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node 2025-12-04 04:26:25,072 - DEBUG - Layer2 route found for VM MAC 02:7c:92:77:d3:92 in Control-node 10.0.0.23 2025-12-04 04:26:25,072 - DEBUG - L2 Label for VM EVPN_VN1_VM2 same between Control-node 10.0.0.23 and Agent, Expected: 13, Seen: 13 2025-12-04 04:26:25,072 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-12-04 04:26:25,204 - DEBUG - Layer2 route found for VM MAC 02:7c:92:77:d3:92 in Control-node 10.0.0.27 2025-12-04 04:26:25,204 - DEBUG - L2 Label for VM EVPN_VN1_VM2 same between Control-node 10.0.0.27 and Agent, Expected: 13, Seen: 13 2025-12-04 04:26:25,204 - DEBUG - Starting all layer2 verification in 10.0.0.33 Control Node 2025-12-04 04:26:25,332 - DEBUG - Layer2 route found for VM MAC 02:7c:92:77:d3:92 in Control-node 10.0.0.33 2025-12-04 04:26:25,332 - DEBUG - L2 Label for VM EVPN_VN1_VM2 same between Control-node 10.0.0.33 and Agent, Expected: 13, Seen: 13 2025-12-04 04:26:25,332 - INFO - Verification in Control-nodes for VM EVPN_VN1_VM2 passed 2025-12-04 04:26:25,332 - DEBUG - Verifying the vm in opserver 2025-12-04 04:26:25,332 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-12-04 04:26:25,332 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat 2025-12-04 04:26:25,349 - DEBUG - Failed to get VM EVPN_VN1_VM2, ID fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 info from Opserver 2025-12-04 04:26:27,350 - DEBUG - Verifying the vm in opserver 2025-12-04 04:26:27,350 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-12-04 04:26:27,350 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat 2025-12-04 04:26:27,367 - DEBUG - Failed to get VM EVPN_VN1_VM2, ID fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 info from Opserver 2025-12-04 04:26:29,368 - DEBUG - Verifying the vm in opserver 2025-12-04 04:26:29,368 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-12-04 04:26:29,368 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat 2025-12-04 04:26:29,387 - DEBUG - Failed to get VM EVPN_VN1_VM2, ID fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 info from Opserver 2025-12-04 04:26:31,387 - DEBUG - Verifying the vm in opserver 2025-12-04 04:26:31,387 - DEBUG - Verifying in collector 10.0.0.23 ... 2025-12-04 04:26:31,387 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat 2025-12-04 04:26:31,405 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-06924132:7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc?flat 2025-12-04 04:26:31,426 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-06924132:7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc?flat 2025-12-04 04:26:31,447 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat 2025-12-04 04:26:31,464 - DEBUG - Verifying vm in vn uve 2025-12-04 04:26:31,464 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestEvpnCasesRestart-06924132:7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc?flat 2025-12-04 04:26:31,489 - DEBUG - VM uve shows interface as {'l2_active': True, 'vn_uuid': 'db97f53b-da08-44b2-bdb7-5bb8c55fbb74', 'gateway': '11.1.1.1', '__T': 1764822389866096, 'is_health_check_active': True, 'port_mirror_enabled': False, 'uuid': '7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc', 'tx_vlan': 65535, 'ip_address': '11.1.1.4', 'rx_vlan': 65535, 'ip4_active': True, 'ip6_address': '::', 'vhostuser_mode': 0, 'virtual_network': 'default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1', 'active': True, 'label': 22, 'vm_uuid': 'fd0cf04c-ae57-485f-a5b0-b04f4aa036a6', 'fixed_ip4_list': ['11.1.1.4'], 'admin_state': True, 'vm_name': 'EVPN_VN1_VM2', 'mac_address': '02:7c:92:77:d3:92', 'ip6_active': False} 2025-12-04 04:26:31,490 - DEBUG - VM uve shows ip address as ['11.1.1.4', '::'] 2025-12-04 04:26:31,490 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:26:31,490 - INFO - Verifying through opserver in 10.0.0.27 2025-12-04 04:26:31,490 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1?flat 2025-12-04 04:26:31,528 - DEBUG - expected vm list fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:26:31,528 - DEBUG - Extracted vm list ['fd0cf04c-ae57-485f-a5b0-b04f4aa036a6'] 2025-12-04 04:26:31,528 - INFO - VM fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is present in default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:26:31,528 - DEBUG - Verifying vm in vrouter uve 2025-12-04 04:26:31,528 - DEBUG - Getting info from collector 10.0.0.23.. 2025-12-04 04:26:31,528 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat 2025-12-04 04:26:31,545 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4840-2'] 2025-12-04 04:26:31,545 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4840-2:Compute:contrail-vrouter-agent:0?flat 2025-12-04 04:26:31,581 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4840-2?flat 2025-12-04 04:26:31,670 - DEBUG - VM fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-2 uve 2025-12-04 04:26:31,670 - DEBUG - Validated that VM EVPN_VN1_VM2 is in Vrouter an-jenkins-deploy-platform-ansible-os-4840-2 UVE 2025-12-04 04:26:31,670 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-12-04 04:26:31,678 - DEBUG - Expected tap interface of VM uuid fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is default-domain:ctest-TestEvpnCasesRestart-06924132:7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc 2025-12-04 04:26:31,678 - DEBUG - Expected VN of VM uuid fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:26:31,678 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4840-2:Compute:contrail-vrouter-agent:0?flat 2025-12-04 04:26:31,714 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4840-2?flat 2025-12-04 04:26:31,746 - DEBUG - VM fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-2 uve 2025-12-04 04:26:31,746 - INFO - tap interface default-domain:ctest-TestEvpnCasesRestart-06924132:7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc of vm fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-2 uve 2025-12-04 04:26:31,746 - DEBUG - Connected network default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 of vm fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is present in vrouter an-jenkins-deploy-platform-ansible-os-4840-2 uve 2025-12-04 04:26:31,746 - DEBUG - Verifying the fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 virtual network link through opserver 10.0.0.27 2025-12-04 04:26:31,746 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-04 04:26:31,759 - DEBUG - VM link and name as {'name': 'fd0cf04c-ae57-485f-a5b0-b04f4aa036a6', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat'} 2025-12-04 04:26:31,759 - DEBUG - Verifying the fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 virtual network link through opserver 10.0.0.33 2025-12-04 04:26:31,759 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-04 04:26:31,774 - DEBUG - VM link and name as {'name': 'fd0cf04c-ae57-485f-a5b0-b04f4aa036a6', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat'} 2025-12-04 04:26:31,774 - DEBUG - Verifying the fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 virtual network link through opserver 10.0.0.23 2025-12-04 04:26:31,774 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-12-04 04:26:31,787 - DEBUG - VM link and name as {'name': 'fd0cf04c-ae57-485f-a5b0-b04f4aa036a6', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6?flat'} 2025-12-04 04:26:31,787 - INFO - VM EVPN_VN1_VM2 validations in Opserver passed 2025-12-04 04:26:31,868 - DEBUG - Executing command: sudo tcpdump -nni tap7c9277d3-92 -U ether src 02:2e:61:90:b3:75 -w /tmp/tap7c9277d3-92_ctest-random-19998639.pcap 2025-12-04 04:26:31,959 - INFO - waiting to get tcpdump started 2025-12-04 04:26:53,095 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=1.11 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.494 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.494/0.799/1.105/0.305 ms') 2025-12-04 04:26:53,095 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 passed 2025-12-04 04:26:53,282 - DEBUG - file transfer done ['/tmp/program.py'] 2025-12-04 04:26:53,282 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-04 04:26:53,282 - DEBUG - Copied file /tmp/tmp1m8g1cg9/program.py to 11.1.1.3 2025-12-04 04:26:54,414 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=1.52 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.468 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.468/0.995/1.522/0.527 ms') 2025-12-04 04:26:54,414 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 passed 2025-12-04 04:26:54,415 - DEBUG - Running remote_cmd, Cmd : python /tmp/program.py, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.22, gateway password: c0ntrail123 2025-12-04 04:26:54,415 - DEBUG - python /tmp/program.py 2025-12-04 04:26:56,538 - DEBUG - WARNING: No route found for IPv6 destination :: (no default route?) .......... Sent 10 packets. 2025-12-04 04:26:56,538 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7c9277d3-92_ctest-random-19998639.pcap ether host 02:7c:92:77:d3:92 | grep -c length 2025-12-04 04:26:56,555 - DEBUG - STDOUT: 0 2025-12-04 04:26:56,555 - DEBUG - STDERR: reading from file /tmp/tap7c9277d3-92_ctest-random-19998639.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-04 04:26:56,555 - WARNING - 0 packets are found in tcpdump output file /tmp/tap7c9277d3-92_ctest-random-19998639.pcap but expected 10 2025-12-04 04:26:58,556 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7c9277d3-92_ctest-random-19998639.pcap ether host 02:7c:92:77:d3:92 | grep -c length 2025-12-04 04:26:58,572 - DEBUG - STDOUT: 0 2025-12-04 04:26:58,573 - DEBUG - STDERR: reading from file /tmp/tap7c9277d3-92_ctest-random-19998639.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-04 04:26:58,573 - WARNING - 0 packets are found in tcpdump output file /tmp/tap7c9277d3-92_ctest-random-19998639.pcap but expected 10 2025-12-04 04:27:00,573 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7c9277d3-92_ctest-random-19998639.pcap ether host 02:7c:92:77:d3:92 | grep -c length 2025-12-04 04:27:00,592 - DEBUG - STDOUT: 0 2025-12-04 04:27:00,592 - DEBUG - STDERR: reading from file /tmp/tap7c9277d3-92_ctest-random-19998639.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-04 04:27:00,592 - WARNING - 0 packets are found in tcpdump output file /tmp/tap7c9277d3-92_ctest-random-19998639.pcap but expected 10 2025-12-04 04:27:02,593 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7c9277d3-92_ctest-random-19998639.pcap ether host 02:7c:92:77:d3:92 | grep -c length 2025-12-04 04:27:02,612 - DEBUG - STDOUT: 0 2025-12-04 04:27:02,612 - DEBUG - STDERR: reading from file /tmp/tap7c9277d3-92_ctest-random-19998639.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-04 04:27:02,613 - WARNING - 0 packets are found in tcpdump output file /tmp/tap7c9277d3-92_ctest-random-19998639.pcap but expected 10 2025-12-04 04:27:04,614 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7c9277d3-92_ctest-random-19998639.pcap ether host 02:7c:92:77:d3:92 | grep -c length 2025-12-04 04:27:04,630 - DEBUG - STDOUT: 0 2025-12-04 04:27:04,631 - DEBUG - STDERR: reading from file /tmp/tap7c9277d3-92_ctest-random-19998639.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-04 04:27:04,631 - WARNING - 0 packets are found in tcpdump output file /tmp/tap7c9277d3-92_ctest-random-19998639.pcap but expected 10 2025-12-04 04:27:06,631 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7c9277d3-92_ctest-random-19998639.pcap ether host 02:7c:92:77:d3:92 | grep -c length 2025-12-04 04:27:06,651 - DEBUG - STDOUT: 0 2025-12-04 04:27:06,651 - DEBUG - STDERR: reading from file /tmp/tap7c9277d3-92_ctest-random-19998639.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-04 04:27:06,651 - WARNING - 0 packets are found in tcpdump output file /tmp/tap7c9277d3-92_ctest-random-19998639.pcap but expected 10 2025-12-04 04:27:08,652 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap7c9277d3-92_ctest-random-19998639.pcap ether host 02:7c:92:77:d3:92 | grep -c length 2025-12-04 04:27:08,668 - DEBUG - STDOUT: 0 2025-12-04 04:27:08,668 - DEBUG - STDERR: reading from file /tmp/tap7c9277d3-92_ctest-random-19998639.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-04 04:27:08,669 - WARNING - 0 packets are found in tcpdump output file /tmp/tap7c9277d3-92_ctest-random-19998639.pcap but expected 10 2025-12-04 04:27:08,669 - INFO - Checking the communication between 2 VM after vrouter restart 2025-12-04 04:27:08,669 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:08,678 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/7c9277d3-92fd-4d13-ad6e-3ee5ecd04fbc 2025-12-04 04:27:08,692 - DEBUG - Requesting: http://10.0.0.23:8082/instance-ip/95ae6efc-429a-47c2-9744-874d2b490978 2025-12-04 04:27:09,874 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=0.810 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.529 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1031ms\r\nrtt min/avg/max/mdev = 0.529/0.669/0.810/0.140 ms') 2025-12-04 04:27:09,874 - INFO - Ping to Metadata IP 169.254.0.3 of VM EVPN_VN1_VM1 passed 2025-12-04 04:27:09,874 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 11.1.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.22, gateway password: c0ntrail123 2025-12-04 04:27:09,875 - DEBUG - ping -s 56 -c 3 -W 1 11.1.1.4 2025-12-04 04:27:12,459 - DEBUG - PING 11.1.1.4 (11.1.1.4) 56(84) bytes of data. 64 bytes from 11.1.1.4: icmp_req=1 ttl=64 time=9.57 ms 64 bytes from 11.1.1.4: icmp_req=2 ttl=64 time=1.29 ms 64 bytes from 11.1.1.4: icmp_req=3 ttl=64 time=1.37 ms --- 11.1.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 1.297/4.082/9.577/3.885 ms 2025-12-04 04:27:12,459 - INFO - Ping to IP 11.1.1.4 from VM EVPN_VN1_VM1 passed 2025-12-04 04:27:12,459 - INFO - Deleting VM EVPN_VN1_VM2 2025-12-04 04:27:12,586 - DEBUG - Verifying in api server 10.0.0.23 2025-12-04 04:27:12,586 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:12,596 - DEBUG - VM ID fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 of VM EVPN_VN1_VM2 is still found in API Server 2025-12-04 04:27:14,597 - DEBUG - Verifying in api server 10.0.0.23 2025-12-04 04:27:14,597 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:14,604 - DEBUG - Response Code: 404 2025-12-04 04:27:14,604 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:14,614 - DEBUG - Response Code: 404 2025-12-04 04:27:14,615 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:14,624 - DEBUG - Response Code: 404 2025-12-04 04:27:14,624 - INFO - VM EVPN_VN1_VM2 is fully removed in API-Server 2025-12-04 04:27:15,624 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-12-04 04:27:15,645 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:15,653 - DEBUG - Response Code: 404 2025-12-04 04:27:16,654 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:16,661 - DEBUG - Response Code: 404 2025-12-04 04:27:17,662 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:17,669 - DEBUG - Response Code: 404 2025-12-04 04:27:18,670 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:18,678 - DEBUG - Response Code: 404 2025-12-04 04:27:19,679 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:19,686 - DEBUG - Response Code: 404 2025-12-04 04:27:20,687 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:20,694 - DEBUG - Response Code: 404 2025-12-04 04:27:20,695 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:20,702 - DEBUG - Response Code: 404 2025-12-04 04:27:21,702 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:21,715 - DEBUG - Response Code: 404 2025-12-04 04:27:22,716 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:22,724 - DEBUG - Response Code: 404 2025-12-04 04:27:23,725 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:23,732 - DEBUG - Response Code: 404 2025-12-04 04:27:24,733 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:24,741 - DEBUG - Response Code: 404 2025-12-04 04:27:25,742 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:25,750 - DEBUG - Response Code: 404 2025-12-04 04:27:25,777 - DEBUG - Validated that vrouter 10.0.0.22 does not have VMs route for VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:27:25,777 - DEBUG - Validated that vrouter 10.0.0.130 does not have VMs route for VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:27:25,778 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:27:25,778 - INFO - VM EVPN_VN1_VM2 is removed in Compute, and routes are removed in all compute nodes 2025-12-04 04:27:26,021 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:26,029 - DEBUG - Response Code: 404 2025-12-04 04:27:27,030 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:27,037 - DEBUG - Response Code: 404 2025-12-04 04:27:28,039 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:28,049 - DEBUG - Response Code: 404 2025-12-04 04:27:29,050 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:29,059 - DEBUG - Response Code: 404 2025-12-04 04:27:30,060 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:30,067 - DEBUG - Response Code: 404 2025-12-04 04:27:31,067 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:31,075 - DEBUG - Response Code: 404 2025-12-04 04:27:31,075 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:31,082 - DEBUG - Response Code: 404 2025-12-04 04:27:32,082 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:32,090 - DEBUG - Response Code: 404 2025-12-04 04:27:33,091 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:33,098 - DEBUG - Response Code: 404 2025-12-04 04:27:34,099 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:34,107 - DEBUG - Response Code: 404 2025-12-04 04:27:35,108 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:35,114 - DEBUG - Response Code: 404 2025-12-04 04:27:36,115 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 2025-12-04 04:27:36,122 - DEBUG - Response Code: 404 2025-12-04 04:27:36,122 - INFO - Routes for VM EVPN_VN1_VM2 is removed in all control-nodes 2025-12-04 04:27:36,122 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4840-2:Compute:contrail-vrouter-agent:0?flat 2025-12-04 04:27:36,171 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4840-2?flat 2025-12-04 04:27:36,205 - DEBUG - VM fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4840-2 uve 2025-12-04 04:27:36,206 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-04 04:27:36,206 - DEBUG - interface for vm fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 deleted from vrouter uve ... 2025-12-04 04:27:36,206 - DEBUG - Deleted interface not in error interface list ... 2025-12-04 04:27:36,206 - DEBUG - Validated that VM fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is removed in Vrouter UVE 2025-12-04 04:27:36,206 - DEBUG - Verifying through opserver in 10.0.0.27 2025-12-04 04:27:36,206 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1?flat 2025-12-04 04:27:36,232 - DEBUG - fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 BM not in default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 uve 2025-12-04 04:27:36,232 - DEBUG - Validated that VM fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is not present in the VN UVE 2025-12-04 04:27:36,232 - DEBUG - Verifying the fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 virtual network link through opserver 10.0.0.27 2025-12-04 04:27:36,232 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-04 04:27:36,244 - INFO - fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 vm uve deleted from opserver 2025-12-04 04:27:36,245 - DEBUG - Verifying the fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 virtual network link through opserver 10.0.0.33 2025-12-04 04:27:36,245 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-04 04:27:36,260 - INFO - fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 vm uve deleted from opserver 2025-12-04 04:27:36,260 - DEBUG - Verifying the fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 virtual network link through opserver 10.0.0.23 2025-12-04 04:27:36,260 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-12-04 04:27:36,274 - INFO - fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 vm uve deleted from opserver 2025-12-04 04:27:36,274 - INFO - Validated that VM fd0cf04c-ae57-485f-a5b0-b04f4aa036a6 is removed from Opserver 2025-12-04 04:27:36,274 - INFO - Deleting VM EVPN_VN1_VM1 2025-12-04 04:27:36,365 - DEBUG - Verifying in api server 10.0.0.23 2025-12-04 04:27:36,365 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:36,375 - DEBUG - VM ID 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc of VM EVPN_VN1_VM1 is still found in API Server 2025-12-04 04:27:38,376 - DEBUG - Verifying in api server 10.0.0.23 2025-12-04 04:27:38,376 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:38,384 - DEBUG - Response Code: 404 2025-12-04 04:27:38,384 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:38,391 - DEBUG - Response Code: 404 2025-12-04 04:27:38,391 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:38,398 - DEBUG - Response Code: 404 2025-12-04 04:27:38,398 - INFO - VM EVPN_VN1_VM1 is fully removed in API-Server 2025-12-04 04:27:38,557 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-12-04 04:27:38,570 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:38,577 - DEBUG - Response Code: 404 2025-12-04 04:27:39,578 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:39,584 - DEBUG - Response Code: 404 2025-12-04 04:27:40,585 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:40,592 - DEBUG - Response Code: 404 2025-12-04 04:27:41,593 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:41,601 - DEBUG - Response Code: 404 2025-12-04 04:27:42,602 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:42,608 - DEBUG - Response Code: 404 2025-12-04 04:27:43,610 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:43,618 - DEBUG - Response Code: 404 2025-12-04 04:27:43,618 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:43,626 - DEBUG - Response Code: 404 2025-12-04 04:27:44,626 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:44,633 - DEBUG - Response Code: 404 2025-12-04 04:27:45,634 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:45,641 - DEBUG - Response Code: 404 2025-12-04 04:27:46,642 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:46,649 - DEBUG - Response Code: 404 2025-12-04 04:27:47,649 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:47,657 - DEBUG - Response Code: 404 2025-12-04 04:27:48,659 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc 2025-12-04 04:27:48,666 - DEBUG - Response Code: 404 2025-12-04 04:27:48,689 - DEBUG - Validated that vrouter 10.0.0.22 does not have VMs route for VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:27:48,701 - DEBUG - Validated that vrouter 10.0.0.130 does not have VMs route for VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:27:48,701 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:27:48,702 - INFO - VM EVPN_VN1_VM1 is removed in Compute, and routes are removed in all compute nodes 2025-12-04 04:27:48,940 - INFO - Routes for VM EVPN_VN1_VM1 is removed in all control-nodes 2025-12-04 04:27:48,940 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4840-1:Compute:contrail-vrouter-agent:0?flat 2025-12-04 04:27:48,977 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4840-1?flat 2025-12-04 04:27:49,009 - DEBUG - VM 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is still present in vrouter an-jenkins-deploy-platform-ansible-os-4840-1 uve 2025-12-04 04:27:49,009 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-04 04:27:49,009 - DEBUG - interface for vm 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc deleted from vrouter uve ... 2025-12-04 04:27:49,009 - DEBUG - Deleted interface not in error interface list ... 2025-12-04 04:27:52,010 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4840-1:Compute:contrail-vrouter-agent:0?flat 2025-12-04 04:27:52,053 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4840-1?flat 2025-12-04 04:27:52,097 - DEBUG - VM 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is not present in vrouter an-jenkins-deploy-platform-ansible-os-4840-1 uve 2025-12-04 04:27:52,097 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-04 04:27:52,097 - DEBUG - interface for vm 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc deleted from vrouter uve ... 2025-12-04 04:27:52,097 - DEBUG - Deleted interface not in error interface list ... 2025-12-04 04:27:52,097 - DEBUG - Validated that VM 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is removed in Vrouter UVE 2025-12-04 04:27:52,097 - DEBUG - Verifying through opserver in 10.0.0.27 2025-12-04 04:27:52,097 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1?flat 2025-12-04 04:27:52,125 - DEBUG - 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc BM not in default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 uve 2025-12-04 04:27:52,126 - DEBUG - Validated that VM 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is not present in the VN UVE 2025-12-04 04:27:52,126 - DEBUG - Verifying the 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc virtual network link through opserver 10.0.0.27 2025-12-04 04:27:52,126 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-04 04:27:52,150 - INFO - 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc vm uve deleted from opserver 2025-12-04 04:27:52,150 - DEBUG - Verifying the 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc virtual network link through opserver 10.0.0.33 2025-12-04 04:27:52,151 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-04 04:27:52,172 - INFO - 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc vm uve deleted from opserver 2025-12-04 04:27:52,172 - DEBUG - Verifying the 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc virtual network link through opserver 10.0.0.23 2025-12-04 04:27:52,172 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines 2025-12-04 04:27:52,189 - INFO - 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc vm uve deleted from opserver 2025-12-04 04:27:52,189 - INFO - Validated that VM 83fc4aaf-8df8-4d6e-bcd3-3a4a273d3bbc is removed from Opserver 2025-12-04 04:27:52,190 - INFO - Deleting VN EVPN-VN1 2025-12-04 04:27:52,446 - DEBUG - Response for deleting network () 2025-12-04 04:27:52,446 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/731d0135-959f-41f8-b28d-8a308053cec2 2025-12-04 04:27:52,454 - DEBUG - Response Code: 404 2025-12-04 04:27:52,455 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-12-04 04:27:52,463 - DEBUG - Requesting: http://10.0.0.27:8082/domain/77f3106b-db9e-49b5-893e-09cac574b890 2025-12-04 04:27:52,531 - DEBUG - Requesting: http://10.0.0.27:8082/project/e9cf5f11-adc9-4409-ae43-9e302251b778 2025-12-04 04:27:52,639 - INFO - Validated that VN EVPN-VN1 is not found in API Server 2025-12-04 04:27:52,651 - DEBUG - VN EVPN-VN1 is not present in Agent 10.0.0.22 2025-12-04 04:27:52,664 - DEBUG - VN EVPN-VN1 is not present in Agent 10.0.0.130 2025-12-04 04:27:52,664 - INFO - Validated that VN EVPN-VN1 is not in any agent 2025-12-04 04:27:52,670 - DEBUG - VRF 2 is not seen in agent 10.0.0.22 2025-12-04 04:27:52,678 - DEBUG - Vrouter 10.0.0.22 does not have vrf 2 for VN EVPN-VN1 2025-12-04 04:27:52,686 - DEBUG - VRF 2 is not seen in agent 10.0.0.130 2025-12-04 04:27:52,694 - DEBUG - Vrouter 10.0.0.130 does not have vrf 2 for VN EVPN-VN1 2025-12-04 04:27:52,694 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestEvpnCasesRestart-06924132:EVPN-VN1 2025-12-04 04:27:52,719 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN EVPN-VN1 info 2025-12-04 04:27:55,168 - DEBUG - Skipping xmpp flap check
2025-12-04 04:27:55,168 - INFO - END TEST : test_with_vxlan_encap_agent_restart : PASSED[0:04:54]
2025-12-04 04:27:55,168 - INFO - -------------------------------------------------------------------------------- 2025-12-04 04:27:56,063 - INFO - Deleted project: ctest-TestEvpnCasesRestart-06924132, ID : e9cf5f11-adc9-4409-ae43-9e302251b778