2025-08-11 17:16:34,623 - INFO - Domain Default found not creating 2025-08-11 17:16:34,754 - INFO - Project ctest-DisablePolicyEcmpIpv6-64293987 not found, creating it 2025-08-11 17:16:35,556 - INFO - Created Project:ctest-DisablePolicyEcmpIpv6-64293987, ID : 721ccc21-1fc4-4fac-b89b-fed4aab29730 2025-08-11 17:16:37,649 - INFO - ================================================================================ 2025-08-11 17:16:37,649 - INFO - STARTING TEST : test_ecmp_with_static_routes 2025-08-11 17:16:37,649 - INFO - TEST DESCRIPTION : Description: Verify disabling policy for ECMP routes with static routes on VM Steps: 1. launch 1 VN and launch 3 VMs in it. 2. create a static route for a new subnet prefix and add this on 2 VMIs. this will create 2 ECMP routes. 3. Disable the policy on all VMIs. 4. Now from 3rd VM send traffic to an IP from static route prefix 5. add new ECMP destinations and verify load is distributed to new destinations too 6. remove ECMP destinations and verify load is distributed to remaining destinations Pass criteria: 1. traffic should go through fine 2. flows should not be created 3. load should be distributed among ecmp routes. 2025-08-11 17:16:38,923 - DEBUG - Nothing to compare xmpp stats {'10.0.0.141': {'10.20.0.192': '0', '10.20.0.193': '0'}, '10.0.0.249': {'10.20.0.193': '0', '10.20.0.192': '0'}} with 2025-08-11 17:16:38,923 - INFO - Address family configured is dual 2025-08-11 17:16:38,923 - INFO - we are setting address family as v6 2025-08-11 17:16:38,923 - INFO - we are setting address family as v6 2025-08-11 17:16:38,923 - INFO - Initial checks done. Running the testcase now 2025-08-11 17:16:38,923 - INFO - 2025-08-11 17:16:38,943 - DEBUG - Services list from nova: [, , , ] 2025-08-11 17:16:38,943 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3816-2', 'an-jenkins-deploy-platform-ansible-os-3816-1']} 2025-08-11 17:16:39,557 - DEBUG - Response for create_network : {'network': {'id': '27a40fdc-5de9-401a-b119-f5e8359e0173', 'name': 'ctest-vn-42089386', 'tenant_id': '721ccc211fc44facb89bfed4aab29730', 'project_id': '721ccc211fc44facb89bfed4aab29730', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-DisablePolicyEcmpIpv6-64293987', 'ctest-vn-42089386'], 'port_security_enabled': True, 'description': ''}} 2025-08-11 17:16:39,806 - DEBUG - Response for create_subnet : {'subnet': {'id': '7e2844f8-cfc4-48d3-b834-7470c42c07aa', 'name': '', 'tenant_id': '721ccc211fc44facb89bfed4aab29730', 'network_id': '27a40fdc-5de9-401a-b119-f5e8359e0173', 'ip_version': 6, 'cidr': '3a1e:a9d0:88e9:bb4e:3694:6fd1::/96', 'allocation_pools': [{'start': '3a1e:a9d0:88e9:bb4e:3694:6fd1:0:2', 'end': '3a1e:a9d0:88e9:bb4e:3694:6fd1:ffff:fffe'}], 'gateway_ip': '3a1e:a9d0:88e9:bb4e:3694:6fd1:0:1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '3a1e:a9d0:88e9:bb4e:3694:6fd1:0:2', 'tags': [], 'project_id': '721ccc211fc44facb89bfed4aab29730'}} 2025-08-11 17:16:40,019 - DEBUG - Response for create_subnet : {'subnet': {'id': '888cdb00-c05e-4159-b260-eab5567be836', 'name': '', 'tenant_id': '721ccc211fc44facb89bfed4aab29730', 'network_id': '27a40fdc-5de9-401a-b119-f5e8359e0173', 'ip_version': 4, 'cidr': '158.63.157.128/26', 'allocation_pools': [{'start': '158.63.157.130', 'end': '158.63.157.190'}], 'gateway_ip': '158.63.157.129', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '158.63.157.130', 'tags': [], 'project_id': '721ccc211fc44facb89bfed4aab29730'}} 2025-08-11 17:16:40,040 - INFO - Created VN ctest-vn-42089386 2025-08-11 17:16:40,053 - DEBUG - VN ctest-vn-42089386 UUID is 27a40fdc-5de9-401a-b119-f5e8359e0173 2025-08-11 17:16:40,138 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-11 17:16:40,431 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-11 17:16:40,457 - DEBUG - Requesting: http://10.0.0.27:8082/domain/4bc87c47-8b22-47dc-8747-cffafaf5f39a 2025-08-11 17:16:40,471 - DEBUG - Requesting: http://10.0.0.27:8082/project/721ccc21-1fc4-4fac-b89b-fed4aab29730 2025-08-11 17:16:40,560 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/27a40fdc-5de9-401a-b119-f5e8359e0173 2025-08-11 17:16:40,573 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/27a40fdc-5de9-401a-b119-f5e8359e0173 2025-08-11 17:16:40,583 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/78576a07-32eb-463f-9423-e4a1220dc226 2025-08-11 17:16:40,591 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/78576a07-32eb-463f-9423-e4a1220dc226 2025-08-11 17:16:40,598 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/43fc1562-2d4b-4e33-8a69-d2f509619b92 2025-08-11 17:16:40,604 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-11 17:16:40,604 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/27a40fdc-5de9-401a-b119-f5e8359e0173 2025-08-11 17:16:40,613 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/78576a07-32eb-463f-9423-e4a1220dc226 2025-08-11 17:16:40,630 - INFO - Verified VN network id 12 for VN 27a40fdc-5de9-401a-b119-f5e8359e0173 2025-08-11 17:16:40,630 - INFO - Verifications in API Server for VN ctest-vn-42089386 passed 2025-08-11 17:16:40,630 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/27a40fdc-5de9-401a-b119-f5e8359e0173 2025-08-11 17:16:40,639 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/78576a07-32eb-463f-9423-e4a1220dc226 2025-08-11 17:16:40,646 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/78576a07-32eb-463f-9423-e4a1220dc226 2025-08-11 17:16:40,654 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/43fc1562-2d4b-4e33-8a69-d2f509619b92 2025-08-11 17:16:40,667 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-DisablePolicyEcmpIpv6-64293987', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2856425502802526234', 'uuid-lslong': '12761501396811776371'}, 'enable': 'true', 'created': '2025-08-11T17:16:39', 'last-modified': '2025-08-11T17:16:39', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.695241'} 2025-08-11 17:16:40,673 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-11 17:16:40,678 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-DisablePolicyEcmpIpv6-64293987', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2856425502802526234', 'uuid-lslong': '12761501396811776371'}, 'enable': 'true', 'created': '2025-08-11T17:16:39', 'last-modified': '2025-08-11T17:16:39', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.709334'} 2025-08-11 17:16:40,684 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-11 17:16:40,690 - DEBUG - Control-node 10.0.0.19 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-DisablePolicyEcmpIpv6-64293987', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2856425502802526234', 'uuid-lslong': '12761501396811776371'}, 'enable': 'true', 'created': '2025-08-11T17:16:39', 'last-modified': '2025-08-11T17:16:39', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.721541'} 2025-08-11 17:16:40,697 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-11 17:16:40,697 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn-42089386 passed 2025-08-11 17:16:40,697 - DEBUG - ====Verifying policy data for ctest-vn-42089386 in API_Server ====== 2025-08-11 17:16:40,697 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-11 17:16:40,703 - DEBUG - Requesting: http://10.0.0.27:8082/domain/4bc87c47-8b22-47dc-8747-cffafaf5f39a 2025-08-11 17:16:40,714 - DEBUG - Requesting: http://10.0.0.27:8082/project/721ccc21-1fc4-4fac-b89b-fed4aab29730 2025-08-11 17:16:40,804 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/27a40fdc-5de9-401a-b119-f5e8359e0173 2025-08-11 17:16:40,815 - DEBUG - =>VN ctest-vn-42089386 has no policy to be verified 2025-08-11 17:16:40,815 - DEBUG - Verifying the vn in opserver 2025-08-11 17:16:40,815 - DEBUG - Verifying the default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 virtual network link through opserver 10.0.0.27 2025-08-11 17:16:40,815 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-11 17:16:41,107 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-11 17:16:41,157 - DEBUG - vn link and name as {'name': 'default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386?flat'} 2025-08-11 17:16:41,157 - DEBUG - Verifying the default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 virtual network link through opserver 10.0.0.23 2025-08-11 17:16:41,157 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-08-11 17:16:41,446 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-08-11 17:16:41,495 - DEBUG - vn link and name as {'name': 'default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386?flat'} 2025-08-11 17:16:41,495 - DEBUG - Verifying the default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 virtual network link through opserver 10.0.0.19 2025-08-11 17:16:41,495 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2025-08-11 17:16:41,785 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2025-08-11 17:16:41,842 - DEBUG - vn link and name as {'name': 'default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386?flat'} 2025-08-11 17:16:41,842 - INFO - Validated that VN default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is found in opserver 2025-08-11 17:16:41,872 - DEBUG - Do not have enough data to verify VN in agent 2025-08-11 17:16:41,881 - DEBUG - VRF ids for VN ctest-vn-42089386: {} 2025-08-11 17:16:41,881 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3816-2', 'an-jenkins-deploy-platform-ansible-os-3816-1']} 2025-08-11 17:16:42,851 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3816-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3816-2) 2025-08-11 17:16:42,851 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3816-2', 'an-jenkins-deploy-platform-ansible-os-3816-1']} 2025-08-11 17:16:44,186 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3816-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3816-1) 2025-08-11 17:16:45,562 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3816-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3816-1) 2025-08-11 17:16:45,562 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3816-2', 'an-jenkins-deploy-platform-ansible-os-3816-1']} 2025-08-11 17:16:46,835 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3816-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3816-2) 2025-08-11 17:16:46,835 - INFO - Waiting for VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 to be up.. 2025-08-11 17:16:46,906 - DEBUG - VM is in ACTIVE state now 2025-08-11 17:16:46,906 - INFO - VM name : ctest-DisablePolicyEcmpIpv6-64293987-42868570 2025-08-11 17:16:46,974 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 ID is 6c4ab3fd-f210-4560-ad92-889e6589344e 2025-08-11 17:16:46,974 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 launched on Node an-jenkins-deploy-platform-ansible-os-3816-1 2025-08-11 17:16:47,052 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-machine/6c4ab3fd-f210-4560-ad92-889e6589344e 2025-08-11 17:16:47,448 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-machine/6c4ab3fd-f210-4560-ad92-889e6589344e 2025-08-11 17:16:47,480 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-machine-interface/efa20373-2d46-47ca-83b7-4dc618b3218f 2025-08-11 17:16:50,743 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-08-11 17:16:50,743 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 failed! 2025-08-11 17:16:50,758 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:1 and allocation pool is NOT set 2025-08-11 17:16:50,758 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 158.63.157.129 and allocation pool is NOT set 2025-08-11 17:16:54,837 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1004ms') 2025-08-11 17:16:54,837 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 failed! 2025-08-11 17:16:54,851 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:1 and allocation pool is NOT set 2025-08-11 17:16:54,852 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 158.63.157.129 and allocation pool is NOT set 2025-08-11 17:16:58,927 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2025-08-11 17:16:58,927 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 failed! 2025-08-11 17:16:58,941 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:1 and allocation pool is NOT set 2025-08-11 17:16:58,941 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 158.63.157.129 and allocation pool is NOT set 2025-08-11 17:17:03,022 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2025-08-11 17:17:03,022 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 failed! 2025-08-11 17:17:03,042 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:1 and allocation pool is NOT set 2025-08-11 17:17:03,042 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 158.63.157.129 and allocation pool is NOT set 2025-08-11 17:17:07,126 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-08-11 17:17:07,126 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 failed! 2025-08-11 17:17:07,140 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:1 and allocation pool is NOT set 2025-08-11 17:17:07,140 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 158.63.157.129 and allocation pool is NOT set 2025-08-11 17:17:11,215 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-08-11 17:17:11,215 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 failed! 2025-08-11 17:17:11,230 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:1 and allocation pool is NOT set 2025-08-11 17:17:11,230 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 158.63.157.129 and allocation pool is NOT set 2025-08-11 17:17:15,317 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-08-11 17:17:15,317 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 failed! 2025-08-11 17:17:15,331 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:1 and allocation pool is NOT set 2025-08-11 17:17:15,331 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 158.63.157.129 and allocation pool is NOT set 2025-08-11 17:17:19,412 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-08-11 17:17:19,412 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 failed! 2025-08-11 17:17:19,426 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:1 and allocation pool is NOT set 2025-08-11 17:17:19,426 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 158.63.157.129 and allocation pool is NOT set 2025-08-11 17:17:23,505 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-08-11 17:17:23,505 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 failed! 2025-08-11 17:17:23,519 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:1 and allocation pool is NOT set 2025-08-11 17:17:23,519 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 158.63.157.129 and allocation pool is NOT set 2025-08-11 17:17:27,597 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-08-11 17:17:27,597 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 failed! 2025-08-11 17:17:27,611 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:1 and allocation pool is NOT set 2025-08-11 17:17:27,611 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 158.63.157.129 and allocation pool is NOT set 2025-08-11 17:17:31,702 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-08-11 17:17:31,702 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 failed! 2025-08-11 17:17:31,719 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:1 and allocation pool is NOT set 2025-08-11 17:17:31,719 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-64293987:ctest-vn-42089386 is 158.63.157.129 and allocation pool is NOT set 2025-08-11 17:17:33,792 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=7.64 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.556 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.556/4.099/7.642/3.543 ms') 2025-08-11 17:17:33,793 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 passed 2025-08-11 17:17:33,908 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-11 17:17:33,908 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-64293987-42868570, IP 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:4, Port 22 2025-08-11 17:17:33,972 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-11 17:17:34,066 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-11 17:17:39,066 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-11 17:17:39,067 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-64293987-42868570, IP 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:4, Port 22 2025-08-11 17:17:39,123 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-11 17:17:39,196 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-11 17:17:44,197 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-11 17:17:44,197 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-64293987-42868570, IP 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:4, Port 22 2025-08-11 17:17:44,264 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-11 17:17:44,336 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-11 17:17:49,337 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-11 17:17:49,337 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-64293987-42868570, IP 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:4, Port 22 2025-08-11 17:17:49,404 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-11 17:17:49,474 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-11 17:17:54,475 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-11 17:17:54,475 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-64293987-42868570, IP 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:4, Port 22 2025-08-11 17:17:54,641 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 is ready for SSH connections 2025-08-11 17:17:54,641 - INFO - Waiting for VM ctest-DisablePolicyEcmpIpv6-64293987-57960530 to be up.. 2025-08-11 17:17:54,709 - DEBUG - VM is in ACTIVE state now 2025-08-11 17:17:54,709 - INFO - VM name : ctest-DisablePolicyEcmpIpv6-64293987-57960530 2025-08-11 17:17:54,781 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-57960530 ID is e7bfc0a7-c8e1-4121-9824-a7fe444ca77a 2025-08-11 17:17:54,781 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-57960530 launched on Node an-jenkins-deploy-platform-ansible-os-3816-1 2025-08-11 17:17:54,845 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-machine/e7bfc0a7-c8e1-4121-9824-a7fe444ca77a 2025-08-11 17:17:54,853 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-machine-interface/ad2279a1-ddad-4f25-aa42-14c69bb82e9a 2025-08-11 17:17:56,048 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.02 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=1.39 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.391/2.204/3.017/0.813 ms') 2025-08-11 17:17:56,048 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-DisablePolicyEcmpIpv6-64293987-57960530 passed 2025-08-11 17:17:56,159 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-11 17:17:56,159 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-64293987-57960530, IP 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:5, Port 22 2025-08-11 17:17:56,310 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-57960530 is ready for SSH connections 2025-08-11 17:17:56,310 - INFO - Waiting for VM ctest-DisablePolicyEcmpIpv6-64293987-80145024 to be up.. 2025-08-11 17:17:56,382 - DEBUG - VM is in ACTIVE state now 2025-08-11 17:17:56,382 - INFO - VM name : ctest-DisablePolicyEcmpIpv6-64293987-80145024 2025-08-11 17:17:56,450 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-80145024 ID is 631e6058-b4b4-4fbe-9b6b-ca22f3788a37 2025-08-11 17:17:56,451 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-80145024 launched on Node an-jenkins-deploy-platform-ansible-os-3816-2 2025-08-11 17:17:56,517 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-machine/631e6058-b4b4-4fbe-9b6b-ca22f3788a37 2025-08-11 17:17:56,526 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-machine-interface/e56390b0-db0a-4694-be18-aa6a774d3f75 2025-08-11 17:17:57,730 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=2.11 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.55 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1000ms\r\nrtt min/avg/max/mdev = 1.550/1.828/2.107/0.278 ms') 2025-08-11 17:17:57,731 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-64293987-80145024 passed 2025-08-11 17:17:57,838 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-11 17:17:57,838 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-64293987-80145024, IP 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:3, Port 22 2025-08-11 17:17:57,994 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-80145024 is ready for SSH connections 2025-08-11 17:17:57,994 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-11 17:17:57,994 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:4 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.141, gateway password: c0ntrail123 2025-08-11 17:17:57,994 - DEBUG - ifconfig -a| grep 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:4 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1 2025-08-11 17:18:15,587 - DEBUG - eth0 2025-08-11 17:18:15,588 - DEBUG - Running remote_cmd, Cmd : sysctl net.ipv6.conf.eth0.accept_dad=0, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.141, gateway password: c0ntrail123 2025-08-11 17:18:15,588 - DEBUG - sysctl net.ipv6.conf.eth0.accept_dad=0 2025-08-11 17:18:16,132 - DEBUG - net.ipv6.conf.eth0.accept_dad = 0 2025-08-11 17:18:16,378 - INFO - Created InterfaceRouteTable ['default-domain', 'ctest-DisablePolicyEcmpIpv6-64293987', 'ctest-intf-rtb-10141542'](UUID b8ae4721-090f-4a26-9452-547eeaeae4d3), prefixes : ['2007:9795:ec9b:2a89:75d6:ba13::/96'] 2025-08-11 17:18:16,458 - INFO - Added intf route table b8ae4721-090f-4a26-9452-547eeaeae4d3 to port efa20373-2d46-47ca-83b7-4dc618b3218f 2025-08-11 17:18:16,459 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.141, gateway password: c0ntrail123 2025-08-11 17:18:16,459 - DEBUG - ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1 2025-08-11 17:18:16,870 - DEBUG - eth0 2025-08-11 17:18:16,870 - DEBUG - Running remote_cmd, Cmd : ifconfig eth0 inet6 add 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.141, gateway password: c0ntrail123 2025-08-11 17:18:16,870 - DEBUG - ifconfig eth0 inet6 add 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c 2025-08-11 17:18:17,352 - DEBUG - None 2025-08-11 17:18:17,352 - DEBUG - Running remote_cmd, Cmd : ifconfig -a, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.141, gateway password: c0ntrail123 2025-08-11 17:18:17,352 - DEBUG - ifconfig -a 2025-08-11 17:18:17,873 - DEBUG - eth0 Link encap:Ethernet HWaddr 02:ef:a2:03:73:2d inet addr:158.63.157.132 Bcast:158.63.157.191 Mask:255.255.255.192 inet6 addr: 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:4/64 Scope:Global inet6 addr: fe80::ef:a2ff:fe03:732d/64 Scope:Link inet6 addr: 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c/128 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:433 errors:0 dropped:0 overruns:0 frame:0 TX packets:357 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:43240 (43.2 KB) TX bytes:33884 (33.8 KB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) 2025-08-11 17:18:17,873 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-11 17:18:17,873 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:5 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.141, gateway password: c0ntrail123 2025-08-11 17:18:17,873 - DEBUG - ifconfig -a| grep 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:5 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1 2025-08-11 17:19:02,539 - DEBUG - eth0 2025-08-11 17:19:02,540 - DEBUG - Running remote_cmd, Cmd : sysctl net.ipv6.conf.eth0.accept_dad=0, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.141, gateway password: c0ntrail123 2025-08-11 17:19:02,540 - DEBUG - sysctl net.ipv6.conf.eth0.accept_dad=0 2025-08-11 17:19:03,042 - DEBUG - net.ipv6.conf.eth0.accept_dad = 0 2025-08-11 17:19:03,262 - INFO - Created InterfaceRouteTable ['default-domain', 'ctest-DisablePolicyEcmpIpv6-64293987', 'ctest-intf-rtb-74951867'](UUID 450a039e-26a8-42c1-a5f9-1b6cc0ba44cc), prefixes : ['2007:9795:ec9b:2a89:75d6:ba13::/96'] 2025-08-11 17:19:03,330 - INFO - Added intf route table 450a039e-26a8-42c1-a5f9-1b6cc0ba44cc to port ad2279a1-ddad-4f25-aa42-14c69bb82e9a 2025-08-11 17:19:03,330 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.141, gateway password: c0ntrail123 2025-08-11 17:19:03,331 - DEBUG - ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1 2025-08-11 17:19:03,738 - DEBUG - eth0 2025-08-11 17:19:03,738 - DEBUG - Running remote_cmd, Cmd : ifconfig eth0 inet6 add 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.141, gateway password: c0ntrail123 2025-08-11 17:19:03,738 - DEBUG - ifconfig eth0 inet6 add 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c 2025-08-11 17:19:04,194 - DEBUG - None 2025-08-11 17:19:04,194 - DEBUG - Running remote_cmd, Cmd : ifconfig -a, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.141, gateway password: c0ntrail123 2025-08-11 17:19:04,194 - DEBUG - ifconfig -a 2025-08-11 17:19:04,683 - DEBUG - eth0 Link encap:Ethernet HWaddr 02:ad:22:79:a1:dd inet addr:158.63.157.133 Bcast:158.63.157.191 Mask:255.255.255.192 inet6 addr: fe80::ad:22ff:fe79:a1dd/64 Scope:Link inet6 addr: 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:5/64 Scope:Global inet6 addr: 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c/128 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:447 errors:0 dropped:0 overruns:0 frame:0 TX packets:402 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:43417 (43.4 KB) TX bytes:36913 (36.9 KB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) 2025-08-11 17:19:04,754 - INFO - Policy DISABLED on VMI e56390b0-db0a-4694-be18-aa6a774d3f75 2025-08-11 17:19:04,822 - INFO - Policy DISABLED on VMI efa20373-2d46-47ca-83b7-4dc618b3218f 2025-08-11 17:19:04,894 - INFO - Policy DISABLED on VMI ad2279a1-ddad-4f25-aa42-14c69bb82e9a 2025-08-11 17:19:04,911 - INFO - ECMP routes found in agent 10.0.0.141, for prefix 2007:9795:ec9b:2a89:75d6:ba13::/96 2025-08-11 17:19:04,911 - INFO - Tap interface tapefa20373-2d found in ecmp routes in agent 10.0.0.141 2025-08-11 17:19:04,911 - INFO - Tap interface tapad2279a1-dd found in ecmp routes in agent 10.0.0.141 2025-08-11 17:19:04,925 - INFO - ECMP routes found in agent 10.0.0.141, for prefix 2007:9795:ec9b:2a89:75d6:ba13::/96 2025-08-11 17:19:05,001 - DEBUG - Executing command: sudo tcpdump -nni tapefa20373-2d -U '(udp and src host 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:3 and dst host 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c and dst port 11000)' -w /tmp/tapefa20373-2d_ctest-random-25446227.pcap 2025-08-11 17:19:05,153 - DEBUG - Executing command: sudo tcpdump -nni tapad2279a1-dd -U '(udp and src host 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:3 and dst host 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c and dst port 11000)' -w /tmp/tapad2279a1-dd_ctest-random-66230864.pcap 2025-08-11 17:19:05,243 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=icmp_error;nc -6 -q 2 -u -p 43962 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c 11000 < icmp_error, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.249, gateway password: c0ntrail123 2025-08-11 17:19:05,244 - DEBUG - dd bs=100 count=1 if=/dev/zero of=icmp_error;nc -6 -q 2 -u -p 43962 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c 11000 < icmp_error 2025-08-11 17:19:54,051 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.000921074 s, 109 kB/s 2025-08-11 17:19:54,052 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapefa20373-2d_ctest-random-25446227.pcap | grep -c length 2025-08-11 17:19:54,064 - DEBUG - STDOUT: 1 2025-08-11 17:19:54,064 - DEBUG - STDERR: reading from file /tmp/tapefa20373-2d_ctest-random-25446227.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-11 17:19:54,064 - INFO - 1 packets are found in tcpdump output as expected 2025-08-11 17:19:54,064 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-08-11 17:19:56,109 - INFO - Tcpdump verification on VM 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:4 passed 2025-08-11 17:19:56,339 - DEBUG - Forward and reverse flows for match {'sip': '3a1e:a9d0:88e9:bb4e:3694:6fd1:0:3', 'dip': '2007:9795:ec9b:2a89:75d6:ba13:66a2:281c', 'sport': '43962', 'dport': '11000', 'proto': '17', 'vrf_id': '2'}: 0, 0 2025-08-11 17:19:56,340 - DEBUG - Flow verification PASSED on node: 10.0.0.141 for VMs - Sender: 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:3, Receiver: 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c, Flow count expected: 0 0, got: 0 0 2025-08-11 17:19:56,340 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-08-11 17:19:58,348 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-08-11 17:20:00,635 - DEBUG - Forward and reverse flows for match {'sip': '3a1e:a9d0:88e9:bb4e:3694:6fd1:0:3', 'dip': '2007:9795:ec9b:2a89:75d6:ba13:66a2:281c', 'sport': '43962', 'dport': '11000', 'proto': '17', 'vrf_id': '2'}: 0, 0 2025-08-11 17:20:00,636 - DEBUG - Flow verification PASSED on node: 10.0.0.249 for VMs - Sender: 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:3, Receiver: 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c, Flow count expected: 0 0, got: 0 0 2025-08-11 17:20:00,636 - INFO - Traffic verification for ECMP passed 2025-08-11 17:20:00,636 - INFO - Waiting for VM ctest-DisablePolicyEcmpIpv6-64293987-13936892 to be up.. 2025-08-11 17:20:00,710 - DEBUG - VM is in ACTIVE state now 2025-08-11 17:20:00,710 - INFO - VM name : ctest-DisablePolicyEcmpIpv6-64293987-13936892 2025-08-11 17:20:00,778 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-13936892 ID is 86dbb55c-1296-4840-9d9e-3f952cf5828c 2025-08-11 17:20:00,778 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-13936892 launched on Node an-jenkins-deploy-platform-ansible-os-3816-2 2025-08-11 17:20:00,850 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-machine/86dbb55c-1296-4840-9d9e-3f952cf5828c 2025-08-11 17:20:00,860 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-machine-interface/debd3350-f331-4ead-945d-4c547e31a56a 2025-08-11 17:20:02,057 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.80 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.473 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.473/2.138/3.803/1.665 ms') 2025-08-11 17:20:02,058 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-DisablePolicyEcmpIpv6-64293987-13936892 passed 2025-08-11 17:20:02,175 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-11 17:20:02,175 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-64293987-13936892, IP 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:6, Port 22 2025-08-11 17:20:02,340 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-64293987-13936892 is ready for SSH connections 2025-08-11 17:20:02,422 - INFO - Policy DISABLED on VMI debd3350-f331-4ead-945d-4c547e31a56a 2025-08-11 17:20:02,422 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-11 17:20:02,422 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:6 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.249, gateway password: c0ntrail123 2025-08-11 17:20:02,423 - DEBUG - ifconfig -a| grep 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:6 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1 2025-08-11 17:20:44,852 - DEBUG - eth0 2025-08-11 17:20:44,852 - DEBUG - Running remote_cmd, Cmd : sysctl net.ipv6.conf.eth0.accept_dad=0, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.249, gateway password: c0ntrail123 2025-08-11 17:20:44,852 - DEBUG - sysctl net.ipv6.conf.eth0.accept_dad=0 2025-08-11 17:20:45,358 - DEBUG - net.ipv6.conf.eth0.accept_dad = 0 2025-08-11 17:20:45,583 - INFO - Created InterfaceRouteTable ['default-domain', 'ctest-DisablePolicyEcmpIpv6-64293987', 'ctest-intf-rtb-01994071'](UUID f8a0e76c-337e-4986-aaf0-348fd8944781), prefixes : ['2007:9795:ec9b:2a89:75d6:ba13::/96'] 2025-08-11 17:20:45,650 - INFO - Added intf route table f8a0e76c-337e-4986-aaf0-348fd8944781 to port debd3350-f331-4ead-945d-4c547e31a56a 2025-08-11 17:20:45,651 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.249, gateway password: c0ntrail123 2025-08-11 17:20:45,651 - DEBUG - ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1 2025-08-11 17:20:46,061 - DEBUG - eth0 2025-08-11 17:20:46,061 - DEBUG - Running remote_cmd, Cmd : ifconfig eth0 inet6 add 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.249, gateway password: c0ntrail123 2025-08-11 17:20:46,061 - DEBUG - ifconfig eth0 inet6 add 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c 2025-08-11 17:20:46,533 - DEBUG - None 2025-08-11 17:20:46,533 - DEBUG - Running remote_cmd, Cmd : ifconfig -a, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.249, gateway password: c0ntrail123 2025-08-11 17:20:46,533 - DEBUG - ifconfig -a 2025-08-11 17:20:47,050 - DEBUG - eth0 Link encap:Ethernet HWaddr 02:de:bd:33:50:f3 inet addr:158.63.157.134 Bcast:158.63.157.191 Mask:255.255.255.192 inet6 addr: fe80::de:bdff:fe33:50f3/64 Scope:Link inet6 addr: 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c/128 Scope:Global inet6 addr: 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:6/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:578 errors:0 dropped:0 overruns:0 frame:0 TX packets:515 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:52133 (52.1 KB) TX bytes:44131 (44.1 KB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) 2025-08-11 17:20:47,051 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-64293987-42868570 2025-08-11 17:20:47,182 - INFO - Removed intf route table b8ae4721-090f-4a26-9452-547eeaeae4d3 from port efa20373-2d46-47ca-83b7-4dc618b3218f 2025-08-11 17:20:47,214 - INFO - Deleted Interface route table b8ae4721-090f-4a26-9452-547eeaeae4d3 2025-08-11 17:20:47,228 - INFO - ECMP routes found in agent 10.0.0.141, for prefix 2007:9795:ec9b:2a89:75d6:ba13::/96 2025-08-11 17:20:47,229 - INFO - Tap interface tapad2279a1-dd found in ecmp routes in agent 10.0.0.141 2025-08-11 17:20:47,244 - INFO - ECMP routes found in agent 10.0.0.249, for prefix 2007:9795:ec9b:2a89:75d6:ba13::/96 2025-08-11 17:20:47,244 - INFO - Tap interface tapdebd3350-f3 found in ecmp routes in agent 10.0.0.249 2025-08-11 17:20:47,323 - DEBUG - Executing command: sudo tcpdump -nni tapad2279a1-dd -U '(udp and src host 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:3 and dst host 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c and dst port 11000)' -w /tmp/tapad2279a1-dd_ctest-random-73859953.pcap 2025-08-11 17:20:47,771 - DEBUG - Executing command: sudo tcpdump -nni tapdebd3350-f3 -U '(udp and src host 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:3 and dst host 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c and dst port 11000)' -w /tmp/tapdebd3350-f3_ctest-random-33646396.pcap 2025-08-11 17:20:48,107 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=icmp_error;nc -6 -q 2 -u -p 48425 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c 11000 < icmp_error, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.249, gateway password: c0ntrail123 2025-08-11 17:20:48,108 - DEBUG - dd bs=100 count=1 if=/dev/zero of=icmp_error;nc -6 -q 2 -u -p 48425 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c 11000 < icmp_error 2025-08-11 17:20:50,621 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.000881993 s, 113 kB/s 2025-08-11 17:20:50,622 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapad2279a1-dd_ctest-random-73859953.pcap | grep -c length 2025-08-11 17:20:50,634 - DEBUG - STDOUT: 0 2025-08-11 17:20:50,634 - DEBUG - STDERR: reading from file /tmp/tapad2279a1-dd_ctest-random-73859953.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-11 17:20:50,634 - WARNING - No packets are found in tcpdump output file /tmp/tapad2279a1-dd_ctest-random-73859953.pcap but expected some packets 2025-08-11 17:20:52,634 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapad2279a1-dd_ctest-random-73859953.pcap | grep -c length 2025-08-11 17:20:52,646 - DEBUG - STDOUT: 0 2025-08-11 17:20:52,647 - DEBUG - STDERR: reading from file /tmp/tapad2279a1-dd_ctest-random-73859953.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-11 17:20:52,647 - WARNING - No packets are found in tcpdump output file /tmp/tapad2279a1-dd_ctest-random-73859953.pcap but expected some packets 2025-08-11 17:20:54,647 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapad2279a1-dd_ctest-random-73859953.pcap | grep -c length 2025-08-11 17:20:54,660 - DEBUG - STDOUT: 0 2025-08-11 17:20:54,660 - DEBUG - STDERR: reading from file /tmp/tapad2279a1-dd_ctest-random-73859953.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-11 17:20:54,660 - WARNING - No packets are found in tcpdump output file /tmp/tapad2279a1-dd_ctest-random-73859953.pcap but expected some packets 2025-08-11 17:20:56,661 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapad2279a1-dd_ctest-random-73859953.pcap | grep -c length 2025-08-11 17:20:56,673 - DEBUG - STDOUT: 0 2025-08-11 17:20:56,673 - DEBUG - STDERR: reading from file /tmp/tapad2279a1-dd_ctest-random-73859953.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-11 17:20:56,673 - WARNING - No packets are found in tcpdump output file /tmp/tapad2279a1-dd_ctest-random-73859953.pcap but expected some packets 2025-08-11 17:20:58,674 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapad2279a1-dd_ctest-random-73859953.pcap | grep -c length 2025-08-11 17:20:58,687 - DEBUG - STDOUT: 0 2025-08-11 17:20:58,687 - DEBUG - STDERR: reading from file /tmp/tapad2279a1-dd_ctest-random-73859953.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-11 17:20:58,687 - WARNING - No packets are found in tcpdump output file /tmp/tapad2279a1-dd_ctest-random-73859953.pcap but expected some packets 2025-08-11 17:21:00,688 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapad2279a1-dd_ctest-random-73859953.pcap | grep -c length 2025-08-11 17:21:00,702 - DEBUG - STDOUT: 0 2025-08-11 17:21:00,702 - DEBUG - STDERR: reading from file /tmp/tapad2279a1-dd_ctest-random-73859953.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-11 17:21:00,702 - WARNING - No packets are found in tcpdump output file /tmp/tapad2279a1-dd_ctest-random-73859953.pcap but expected some packets 2025-08-11 17:21:02,702 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapad2279a1-dd_ctest-random-73859953.pcap | grep -c length 2025-08-11 17:21:02,715 - DEBUG - STDOUT: 0 2025-08-11 17:21:02,715 - DEBUG - STDERR: reading from file /tmp/tapad2279a1-dd_ctest-random-73859953.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-11 17:21:02,715 - WARNING - No packets are found in tcpdump output file /tmp/tapad2279a1-dd_ctest-random-73859953.pcap but expected some packets 2025-08-11 17:21:02,715 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapdebd3350-f3_ctest-random-33646396.pcap | grep -c length 2025-08-11 17:21:02,728 - DEBUG - STDOUT: 1 2025-08-11 17:21:02,728 - DEBUG - STDERR: reading from file /tmp/tapdebd3350-f3_ctest-random-33646396.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-11 17:21:02,728 - INFO - 1 packets are found in tcpdump output as expected 2025-08-11 17:21:02,728 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-08-11 17:21:04,772 - INFO - Tcpdump verification on VM 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:6 passed 2025-08-11 17:21:05,057 - DEBUG - Forward and reverse flows for match {'sip': '3a1e:a9d0:88e9:bb4e:3694:6fd1:0:3', 'dip': '2007:9795:ec9b:2a89:75d6:ba13:66a2:281c', 'sport': '48425', 'dport': '11000', 'proto': '17', 'vrf_id': '2'}: 0, 0 2025-08-11 17:21:05,057 - DEBUG - Flow verification PASSED on node: 10.0.0.249 for VMs - Sender: 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:3, Receiver: 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c, Flow count expected: 0 0, got: 0 0 2025-08-11 17:21:05,057 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-08-11 17:21:07,064 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-08-11 17:21:09,381 - DEBUG - Forward and reverse flows for match {'sip': '3a1e:a9d0:88e9:bb4e:3694:6fd1:0:3', 'dip': '2007:9795:ec9b:2a89:75d6:ba13:66a2:281c', 'sport': '48425', 'dport': '11000', 'proto': '17', 'vrf_id': '2'}: 0, 0 2025-08-11 17:21:09,381 - DEBUG - Flow verification PASSED on node: 10.0.0.249 for VMs - Sender: 3a1e:a9d0:88e9:bb4e:3694:6fd1:0:3, Receiver: 2007:9795:ec9b:2a89:75d6:ba13:66a2:281c, Flow count expected: 0 0, got: 0 0 2025-08-11 17:21:09,381 - INFO - Traffic verification for ECMP passed 2025-08-11 17:21:09,381 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-64293987-13936892 2025-08-11 17:21:09,518 - INFO - Removed intf route table f8a0e76c-337e-4986-aaf0-348fd8944781 from port debd3350-f331-4ead-945d-4c547e31a56a 2025-08-11 17:21:09,549 - INFO - Deleted Interface route table f8a0e76c-337e-4986-aaf0-348fd8944781 2025-08-11 17:21:09,549 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-64293987-57960530 2025-08-11 17:21:09,686 - INFO - Removed intf route table 450a039e-26a8-42c1-a5f9-1b6cc0ba44cc from port ad2279a1-ddad-4f25-aa42-14c69bb82e9a 2025-08-11 17:21:09,717 - INFO - Deleted Interface route table 450a039e-26a8-42c1-a5f9-1b6cc0ba44cc 2025-08-11 17:21:09,717 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-64293987-80145024 2025-08-11 17:21:09,778 - INFO - Deleting VN ctest-vn-42089386 2025-08-11 17:21:09,815 - DEBUG - VN 27a40fdc-5de9-401a-b119-f5e8359e0173 still in use: Unable to complete operation on network 27a40fdc-5de9-401a-b119-f5e8359e0173. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-804375cb-e645-43e7-b802-2fac70a48c2f'] 2025-08-11 17:21:09,815 - WARNING - Deleting VN ctest-vn-42089386 failed..Will retry 2025-08-11 17:21:11,991 - DEBUG - Response for deleting network () 2025-08-11 17:21:11,991 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/78576a07-32eb-463f-9423-e4a1220dc226 2025-08-11 17:21:11,998 - DEBUG - Response Code: 404 2025-08-11 17:21:11,998 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-11 17:21:12,009 - DEBUG - Requesting: http://10.0.0.27:8082/domain/4bc87c47-8b22-47dc-8747-cffafaf5f39a 2025-08-11 17:21:12,071 - DEBUG - Requesting: http://10.0.0.27:8082/project/721ccc21-1fc4-4fac-b89b-fed4aab29730 2025-08-11 17:21:12,157 - INFO - Validated that VN ctest-vn-42089386 is not found in API Server 2025-08-11 17:21:12,168 - DEBUG - VN ctest-vn-42089386 is not present in Agent 10.0.0.141 2025-08-11 17:21:12,179 - DEBUG - VN ctest-vn-42089386 is not present in Agent 10.0.0.249 2025-08-11 17:21:12,179 - INFO - Validated that VN ctest-vn-42089386 is not in any agent 2025-08-11 17:21:12,210 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn-42089386 info 2025-08-11 17:21:13,455 - DEBUG - No XMPP flaps were noticed during the test
2025-08-11 17:21:13,455 - INFO - END TEST : test_ecmp_with_static_routes : PASSED[0:04:36]
2025-08-11 17:21:13,455 - INFO - -------------------------------------------------------------------------------- 2025-08-11 17:21:14,199 - INFO - Deleted project: ctest-DisablePolicyEcmpIpv6-64293987, ID : 721ccc21-1fc4-4fac-b89b-fed4aab29730