2025-11-27 03:16:17,052 - INFO - Domain Default found not creating 2025-11-27 03:16:17,288 - INFO - Project ctest-DisablePolicyEcmpIpv6-67056033 not found, creating it 2025-11-27 03:16:17,784 - INFO - Created Project:ctest-DisablePolicyEcmpIpv6-67056033, ID : 633eec2a-4e6c-43e6-af7e-507d29215bf6 2025-11-27 03:16:19,280 - INFO - ================================================================================ 2025-11-27 03:16:19,280 - INFO - STARTING TEST : test_ecmp_with_static_routes 2025-11-27 03:16:19,280 - INFO - TEST DESCRIPTION : Description: Verify disabling policy for ECMP routes with static routes on VM Steps: 1. launch 1 VN and launch 3 VMs in it. 2. create a static route for a new subnet prefix and add this on 2 VMIs. this will create 2 ECMP routes. 3. Disable the policy on all VMIs. 4. Now from 3rd VM send traffic to an IP from static route prefix 5. add new ECMP destinations and verify load is distributed to new destinations too 6. remove ECMP destinations and verify load is distributed to remaining destinations Pass criteria: 1. traffic should go through fine 2. flows should not be created 3. load should be distributed among ecmp routes. 2025-11-27 03:16:20,550 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.193': '0', '10.20.0.192': '0'}, '10.0.0.130': {'10.20.0.192': '0', '10.20.0.129': '0'}} with 2025-11-27 03:16:20,551 - INFO - Address family configured is dual 2025-11-27 03:16:20,551 - INFO - we are setting address family as v6 2025-11-27 03:16:20,551 - INFO - we are setting address family as v6 2025-11-27 03:16:20,551 - INFO - Initial checks done. Running the testcase now 2025-11-27 03:16:20,551 - INFO - 2025-11-27 03:16:20,572 - DEBUG - Services list from nova: [, , , ] 2025-11-27 03:16:20,573 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-4820-2', 'an-jenkins-deploy-platform-ansible-os-4820-1']} 2025-11-27 03:16:21,250 - DEBUG - Response for create_network : {'network': {'id': '7261bb8c-8840-4bf9-a564-b655c5f649d0', 'name': 'ctest-vn-43940798', 'tenant_id': '633eec2a4e6c43e6af7e507d29215bf6', 'project_id': '633eec2a4e6c43e6af7e507d29215bf6', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-DisablePolicyEcmpIpv6-67056033', 'ctest-vn-43940798'], 'port_security_enabled': True, 'description': ''}} 2025-11-27 03:16:21,527 - DEBUG - Response for create_subnet : {'subnet': {'id': '2290cec0-e669-4b36-b138-24980855fc71', 'name': '', 'tenant_id': '633eec2a4e6c43e6af7e507d29215bf6', 'network_id': '7261bb8c-8840-4bf9-a564-b655c5f649d0', 'ip_version': 6, 'cidr': '38b3:f805:cfff:3f05:a32a:4df9::/96', 'allocation_pools': [{'start': '38b3:f805:cfff:3f05:a32a:4df9:0:2', 'end': '38b3:f805:cfff:3f05:a32a:4df9:ffff:fffe'}], 'gateway_ip': '38b3:f805:cfff:3f05:a32a:4df9:0:1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '38b3:f805:cfff:3f05:a32a:4df9:0:2', 'tags': [], 'project_id': '633eec2a4e6c43e6af7e507d29215bf6'}} 2025-11-27 03:16:21,719 - DEBUG - Response for create_subnet : {'subnet': {'id': '1579f4f0-728b-4bc0-b53d-9e4de5c836d8', 'name': '', 'tenant_id': '633eec2a4e6c43e6af7e507d29215bf6', 'network_id': '7261bb8c-8840-4bf9-a564-b655c5f649d0', 'ip_version': 4, 'cidr': '68.78.88.192/26', 'allocation_pools': [{'start': '68.78.88.194', 'end': '68.78.88.254'}], 'gateway_ip': '68.78.88.193', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '68.78.88.194', 'tags': [], 'project_id': '633eec2a4e6c43e6af7e507d29215bf6'}} 2025-11-27 03:16:21,745 - INFO - Created VN ctest-vn-43940798 2025-11-27 03:16:21,755 - DEBUG - VN ctest-vn-43940798 UUID is 7261bb8c-8840-4bf9-a564-b655c5f649d0 2025-11-27 03:16:21,855 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-11-27 03:16:22,148 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-11-27 03:16:22,176 - DEBUG - Requesting: http://10.0.0.27:8082/domain/ea95f9c6-3b7f-4f10-9ecb-80826711f59a 2025-11-27 03:16:22,187 - DEBUG - Requesting: http://10.0.0.27:8082/project/633eec2a-4e6c-43e6-af7e-507d29215bf6 2025-11-27 03:16:22,280 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/7261bb8c-8840-4bf9-a564-b655c5f649d0 2025-11-27 03:16:22,292 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/7261bb8c-8840-4bf9-a564-b655c5f649d0 2025-11-27 03:16:22,301 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/eb87cc2f-0c7a-4931-94b1-dab0e2deb176 2025-11-27 03:16:22,308 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/eb87cc2f-0c7a-4931-94b1-dab0e2deb176 2025-11-27 03:16:22,314 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/e2931d5e-d13b-4422-bdb7-71de6b71fabe 2025-11-27 03:16:22,320 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-27 03:16:22,320 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/7261bb8c-8840-4bf9-a564-b655c5f649d0 2025-11-27 03:16:22,329 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/eb87cc2f-0c7a-4931-94b1-dab0e2deb176 2025-11-27 03:16:22,346 - INFO - Verified VN network id 14 for VN 7261bb8c-8840-4bf9-a564-b655c5f649d0 2025-11-27 03:16:22,346 - INFO - Verifications in API Server for VN ctest-vn-43940798 passed 2025-11-27 03:16:22,346 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/7261bb8c-8840-4bf9-a564-b655c5f649d0 2025-11-27 03:16:22,357 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/eb87cc2f-0c7a-4931-94b1-dab0e2deb176 2025-11-27 03:16:22,363 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/eb87cc2f-0c7a-4931-94b1-dab0e2deb176 2025-11-27 03:16:22,369 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/e2931d5e-d13b-4422-bdb7-71de6b71fabe 2025-11-27 03:16:22,381 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-DisablePolicyEcmpIpv6-67056033', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '8242075005320449017', 'uuid-lslong': '11917850993438902736'}, 'enable': 'true', 'created': '2025-11-27T03:16:21', 'last-modified': '2025-11-27T03:16:21', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.704444'} 2025-11-27 03:16:22,386 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-27 03:16:22,391 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-DisablePolicyEcmpIpv6-67056033', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '8242075005320449017', 'uuid-lslong': '11917850993438902736'}, 'enable': 'true', 'created': '2025-11-27T03:16:21', 'last-modified': '2025-11-27T03:16:21', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.714811'} 2025-11-27 03:16:22,397 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-27 03:16:22,402 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-DisablePolicyEcmpIpv6-67056033', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '8242075005320449017', 'uuid-lslong': '11917850993438902736'}, 'enable': 'true', 'created': '2025-11-27T03:16:21', 'last-modified': '2025-11-27T03:16:21', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.723163'} 2025-11-27 03:16:22,408 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-11-27 03:16:22,408 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn-43940798 passed 2025-11-27 03:16:22,408 - DEBUG - ====Verifying policy data for ctest-vn-43940798 in API_Server ====== 2025-11-27 03:16:22,408 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-11-27 03:16:22,415 - DEBUG - Requesting: http://10.0.0.27:8082/domain/ea95f9c6-3b7f-4f10-9ecb-80826711f59a 2025-11-27 03:16:22,428 - DEBUG - Requesting: http://10.0.0.27:8082/project/633eec2a-4e6c-43e6-af7e-507d29215bf6 2025-11-27 03:16:22,521 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/7261bb8c-8840-4bf9-a564-b655c5f649d0 2025-11-27 03:16:22,531 - DEBUG - =>VN ctest-vn-43940798 has no policy to be verified 2025-11-27 03:16:22,531 - DEBUG - Verifying the vn in opserver 2025-11-27 03:16:22,531 - DEBUG - Verifying the default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 virtual network link through opserver 10.0.0.27 2025-11-27 03:16:22,532 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-11-27 03:16:22,832 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-11-27 03:16:22,913 - DEBUG - vn link and name as {'name': 'default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798?flat'} 2025-11-27 03:16:22,914 - DEBUG - Verifying the default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 virtual network link through opserver 10.0.0.33 2025-11-27 03:16:22,914 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-11-27 03:16:23,244 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-11-27 03:16:23,305 - DEBUG - vn link and name as {'name': 'default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798?flat'} 2025-11-27 03:16:23,305 - DEBUG - Verifying the default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 virtual network link through opserver 10.0.0.23 2025-11-27 03:16:23,305 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-11-27 03:16:23,610 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-11-27 03:16:23,668 - DEBUG - vn link and name as {'name': 'default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798?flat'} 2025-11-27 03:16:23,668 - INFO - Validated that VN default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is found in opserver 2025-11-27 03:16:23,688 - DEBUG - Do not have enough data to verify VN in agent 2025-11-27 03:16:23,696 - DEBUG - VRF ids for VN ctest-vn-43940798: {} 2025-11-27 03:16:23,696 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-4820-2', 'an-jenkins-deploy-platform-ansible-os-4820-1']} 2025-11-27 03:16:24,788 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4820-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4820-2) 2025-11-27 03:16:24,789 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-4820-2', 'an-jenkins-deploy-platform-ansible-os-4820-1']} 2025-11-27 03:16:26,129 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4820-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4820-1) 2025-11-27 03:16:27,593 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4820-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4820-1) 2025-11-27 03:16:27,594 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-4820-2', 'an-jenkins-deploy-platform-ansible-os-4820-1']} 2025-11-27 03:16:28,978 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4820-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4820-2) 2025-11-27 03:16:28,978 - INFO - Waiting for VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 to be up.. 2025-11-27 03:16:29,050 - DEBUG - VM is in ACTIVE state now 2025-11-27 03:16:29,050 - INFO - VM name : ctest-DisablePolicyEcmpIpv6-67056033-63678874 2025-11-27 03:16:29,134 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 ID is 0a61247a-0a21-456e-80c5-f53c5b08e149 2025-11-27 03:16:29,134 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 launched on Node an-jenkins-deploy-platform-ansible-os-4820-1 2025-11-27 03:16:29,224 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/0a61247a-0a21-456e-80c5-f53c5b08e149 2025-11-27 03:16:29,628 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/0a61247a-0a21-456e-80c5-f53c5b08e149 2025-11-27 03:16:29,698 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/fc6b1bf8-8e6d-4d2d-9d9f-e5af401628ba 2025-11-27 03:16:32,899 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1024ms') 2025-11-27 03:16:32,899 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 failed! 2025-11-27 03:16:32,915 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 38b3:f805:cfff:3f05:a32a:4df9:0:1 and allocation pool is NOT set 2025-11-27 03:16:32,915 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 68.78.88.193 and allocation pool is NOT set 2025-11-27 03:16:36,996 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-11-27 03:16:36,996 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 failed! 2025-11-27 03:16:37,011 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 38b3:f805:cfff:3f05:a32a:4df9:0:1 and allocation pool is NOT set 2025-11-27 03:16:37,012 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 68.78.88.193 and allocation pool is NOT set 2025-11-27 03:16:41,093 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-11-27 03:16:41,093 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 failed! 2025-11-27 03:16:41,107 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 38b3:f805:cfff:3f05:a32a:4df9:0:1 and allocation pool is NOT set 2025-11-27 03:16:41,107 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 68.78.88.193 and allocation pool is NOT set 2025-11-27 03:16:45,190 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-11-27 03:16:45,190 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 failed! 2025-11-27 03:16:45,208 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 38b3:f805:cfff:3f05:a32a:4df9:0:1 and allocation pool is NOT set 2025-11-27 03:16:45,208 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 68.78.88.193 and allocation pool is NOT set 2025-11-27 03:16:49,289 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-11-27 03:16:49,289 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 failed! 2025-11-27 03:16:49,303 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 38b3:f805:cfff:3f05:a32a:4df9:0:1 and allocation pool is NOT set 2025-11-27 03:16:49,303 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 68.78.88.193 and allocation pool is NOT set 2025-11-27 03:16:53,382 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-11-27 03:16:53,383 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 failed! 2025-11-27 03:16:53,398 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 38b3:f805:cfff:3f05:a32a:4df9:0:1 and allocation pool is NOT set 2025-11-27 03:16:53,398 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 68.78.88.193 and allocation pool is NOT set 2025-11-27 03:16:57,484 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-11-27 03:16:57,484 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 failed! 2025-11-27 03:16:57,496 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 38b3:f805:cfff:3f05:a32a:4df9:0:1 and allocation pool is NOT set 2025-11-27 03:16:57,496 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 68.78.88.193 and allocation pool is NOT set 2025-11-27 03:17:01,578 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-11-27 03:17:01,578 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 failed! 2025-11-27 03:17:01,592 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 38b3:f805:cfff:3f05:a32a:4df9:0:1 and allocation pool is NOT set 2025-11-27 03:17:01,593 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 68.78.88.193 and allocation pool is NOT set 2025-11-27 03:17:05,674 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-11-27 03:17:05,674 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 failed! 2025-11-27 03:17:05,691 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 38b3:f805:cfff:3f05:a32a:4df9:0:1 and allocation pool is NOT set 2025-11-27 03:17:05,691 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 68.78.88.193 and allocation pool is NOT set 2025-11-27 03:17:09,766 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-11-27 03:17:09,766 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 failed! 2025-11-27 03:17:09,778 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 38b3:f805:cfff:3f05:a32a:4df9:0:1 and allocation pool is NOT set 2025-11-27 03:17:09,779 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 68.78.88.193 and allocation pool is NOT set 2025-11-27 03:17:13,858 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-11-27 03:17:13,859 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 failed! 2025-11-27 03:17:13,872 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 38b3:f805:cfff:3f05:a32a:4df9:0:1 and allocation pool is NOT set 2025-11-27 03:17:13,872 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 68.78.88.193 and allocation pool is NOT set 2025-11-27 03:17:17,959 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-11-27 03:17:17,959 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 failed! 2025-11-27 03:17:17,974 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 38b3:f805:cfff:3f05:a32a:4df9:0:1 and allocation pool is NOT set 2025-11-27 03:17:17,974 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 68.78.88.193 and allocation pool is NOT set 2025-11-27 03:17:22,054 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-11-27 03:17:22,054 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 failed! 2025-11-27 03:17:22,068 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 38b3:f805:cfff:3f05:a32a:4df9:0:1 and allocation pool is NOT set 2025-11-27 03:17:22,068 - DEBUG - Gateway for vn default-domain:ctest-DisablePolicyEcmpIpv6-67056033:ctest-vn-43940798 is 68.78.88.193 and allocation pool is NOT set 2025-11-27 03:17:24,127 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=10.1 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.09 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.093/5.605/10.118/4.512 ms') 2025-11-27 03:17:24,127 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 passed 2025-11-27 03:17:24,192 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-27 03:17:24,192 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-67056033-63678874, IP 38b3:f805:cfff:3f05:a32a:4df9:0:4, Port 22 2025-11-27 03:17:24,261 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-27 03:17:24,343 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-27 03:17:29,344 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-27 03:17:29,344 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-67056033-63678874, IP 38b3:f805:cfff:3f05:a32a:4df9:0:4, Port 22 2025-11-27 03:17:29,406 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-27 03:17:29,493 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-27 03:17:34,495 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-27 03:17:34,495 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-67056033-63678874, IP 38b3:f805:cfff:3f05:a32a:4df9:0:4, Port 22 2025-11-27 03:17:34,560 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-27 03:17:34,664 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-27 03:17:39,665 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-27 03:17:39,665 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-67056033-63678874, IP 38b3:f805:cfff:3f05:a32a:4df9:0:4, Port 22 2025-11-27 03:17:39,732 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-11-27 03:17:39,818 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 is NOT ready for SSH connections, VM status: ACTIVE 2025-11-27 03:17:44,819 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-27 03:17:44,819 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-67056033-63678874, IP 38b3:f805:cfff:3f05:a32a:4df9:0:4, Port 22 2025-11-27 03:17:44,989 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 is ready for SSH connections 2025-11-27 03:17:44,989 - INFO - Waiting for VM ctest-DisablePolicyEcmpIpv6-67056033-10758646 to be up.. 2025-11-27 03:17:45,072 - DEBUG - VM is in ACTIVE state now 2025-11-27 03:17:45,072 - INFO - VM name : ctest-DisablePolicyEcmpIpv6-67056033-10758646 2025-11-27 03:17:45,147 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-10758646 ID is 179ae8b6-49ee-4a55-bfbb-1b196d6d4124 2025-11-27 03:17:45,147 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-10758646 launched on Node an-jenkins-deploy-platform-ansible-os-4820-1 2025-11-27 03:17:45,226 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/179ae8b6-49ee-4a55-bfbb-1b196d6d4124 2025-11-27 03:17:45,234 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/747aeae1-6287-42c5-a47e-1160c3aa7fb0 2025-11-27 03:17:46,398 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=2.33 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=2.07 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 2.072/2.199/2.326/0.127 ms') 2025-11-27 03:17:46,398 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-DisablePolicyEcmpIpv6-67056033-10758646 passed 2025-11-27 03:17:46,464 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-27 03:17:46,465 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-67056033-10758646, IP 38b3:f805:cfff:3f05:a32a:4df9:0:5, Port 22 2025-11-27 03:17:46,633 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-10758646 is ready for SSH connections 2025-11-27 03:17:46,633 - INFO - Waiting for VM ctest-DisablePolicyEcmpIpv6-67056033-87001045 to be up.. 2025-11-27 03:17:46,751 - DEBUG - VM is in ACTIVE state now 2025-11-27 03:17:46,751 - INFO - VM name : ctest-DisablePolicyEcmpIpv6-67056033-87001045 2025-11-27 03:17:46,833 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-87001045 ID is f8ce35d8-dcd6-41d2-921b-899895f05990 2025-11-27 03:17:46,833 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-87001045 launched on Node an-jenkins-deploy-platform-ansible-os-4820-2 2025-11-27 03:17:46,919 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/f8ce35d8-dcd6-41d2-921b-899895f05990 2025-11-27 03:17:46,929 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/b79be315-aa43-44b2-8fa6-a8ec5859e0f3 2025-11-27 03:17:48,084 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=3.30 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=2.11 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 2.109/2.702/3.296/0.593 ms') 2025-11-27 03:17:48,084 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-DisablePolicyEcmpIpv6-67056033-87001045 passed 2025-11-27 03:17:48,156 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-27 03:17:48,156 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-67056033-87001045, IP 38b3:f805:cfff:3f05:a32a:4df9:0:3, Port 22 2025-11-27 03:17:48,323 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-87001045 is ready for SSH connections 2025-11-27 03:17:48,323 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-27 03:17:48,323 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep 38b3:f805:cfff:3f05:a32a:4df9:0:4 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123 2025-11-27 03:17:48,323 - DEBUG - ifconfig -a| grep 38b3:f805:cfff:3f05:a32a:4df9:0:4 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1 2025-11-27 03:18:07,300 - DEBUG - eth0 2025-11-27 03:18:07,300 - DEBUG - Running remote_cmd, Cmd : sysctl net.ipv6.conf.eth0.accept_dad=0, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123 2025-11-27 03:18:07,300 - DEBUG - sysctl net.ipv6.conf.eth0.accept_dad=0 2025-11-27 03:18:07,845 - DEBUG - net.ipv6.conf.eth0.accept_dad = 0 2025-11-27 03:18:08,052 - INFO - Created InterfaceRouteTable ['default-domain', 'ctest-DisablePolicyEcmpIpv6-67056033', 'ctest-intf-rtb-75074317'](UUID 4df947a3-2d57-4643-95ef-3bbd8bd244cd), prefixes : ['3912:6fe9:9a3d:f0a5:da40:767a::/96'] 2025-11-27 03:18:08,116 - INFO - Added intf route table 4df947a3-2d57-4643-95ef-3bbd8bd244cd to port fc6b1bf8-8e6d-4d2d-9d9f-e5af401628ba 2025-11-27 03:18:08,116 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123 2025-11-27 03:18:08,116 - DEBUG - ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1 2025-11-27 03:18:08,590 - DEBUG - eth0 2025-11-27 03:18:08,590 - DEBUG - Running remote_cmd, Cmd : ifconfig eth0 inet6 add 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123 2025-11-27 03:18:08,590 - DEBUG - ifconfig eth0 inet6 add 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292 2025-11-27 03:18:09,121 - DEBUG - None 2025-11-27 03:18:09,121 - DEBUG - Running remote_cmd, Cmd : ifconfig -a, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123 2025-11-27 03:18:09,121 - DEBUG - ifconfig -a 2025-11-27 03:18:09,693 - DEBUG - eth0 Link encap:Ethernet HWaddr 02:fc:6b:1b:f8:8e inet addr:68.78.88.196 Bcast:68.78.88.255 Mask:255.255.255.192 inet6 addr: fe80::fc:6bff:fe1b:f88e/64 Scope:Link inet6 addr: 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292/128 Scope:Global inet6 addr: 38b3:f805:cfff:3f05:a32a:4df9:0:4/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:444 errors:0 dropped:0 overruns:0 frame:0 TX packets:356 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:43999 (43.9 KB) TX bytes:33856 (33.8 KB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) 2025-11-27 03:18:09,693 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-27 03:18:09,693 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep 38b3:f805:cfff:3f05:a32a:4df9:0:5 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123 2025-11-27 03:18:09,693 - DEBUG - ifconfig -a| grep 38b3:f805:cfff:3f05:a32a:4df9:0:5 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1 2025-11-27 03:18:48,987 - DEBUG - eth0 2025-11-27 03:18:48,987 - DEBUG - Running remote_cmd, Cmd : sysctl net.ipv6.conf.eth0.accept_dad=0, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123 2025-11-27 03:18:48,987 - DEBUG - sysctl net.ipv6.conf.eth0.accept_dad=0 2025-11-27 03:18:49,542 - DEBUG - net.ipv6.conf.eth0.accept_dad = 0 2025-11-27 03:18:49,756 - INFO - Created InterfaceRouteTable ['default-domain', 'ctest-DisablePolicyEcmpIpv6-67056033', 'ctest-intf-rtb-66186678'](UUID ac2f06b2-af30-4575-870a-2b3a4ed37051), prefixes : ['3912:6fe9:9a3d:f0a5:da40:767a::/96'] 2025-11-27 03:18:49,824 - INFO - Added intf route table ac2f06b2-af30-4575-870a-2b3a4ed37051 to port 747aeae1-6287-42c5-a47e-1160c3aa7fb0 2025-11-27 03:18:49,824 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123 2025-11-27 03:18:49,824 - DEBUG - ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1 2025-11-27 03:18:50,327 - DEBUG - eth0 2025-11-27 03:18:50,327 - DEBUG - Running remote_cmd, Cmd : ifconfig eth0 inet6 add 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123 2025-11-27 03:18:50,327 - DEBUG - ifconfig eth0 inet6 add 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292 2025-11-27 03:18:50,825 - DEBUG - None 2025-11-27 03:18:50,826 - DEBUG - Running remote_cmd, Cmd : ifconfig -a, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123 2025-11-27 03:18:50,826 - DEBUG - ifconfig -a 2025-11-27 03:18:51,356 - DEBUG - eth0 Link encap:Ethernet HWaddr 02:74:7a:ea:e1:62 inet addr:68.78.88.197 Bcast:68.78.88.255 Mask:255.255.255.192 inet6 addr: 38b3:f805:cfff:3f05:a32a:4df9:0:5/64 Scope:Global inet6 addr: 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292/128 Scope:Global inet6 addr: fe80::74:7aff:feea:e162/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:465 errors:0 dropped:0 overruns:0 frame:0 TX packets:406 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:44761 (44.7 KB) TX bytes:37066 (37.0 KB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) 2025-11-27 03:18:51,424 - INFO - Policy DISABLED on VMI b79be315-aa43-44b2-8fa6-a8ec5859e0f3 2025-11-27 03:18:51,492 - INFO - Policy DISABLED on VMI fc6b1bf8-8e6d-4d2d-9d9f-e5af401628ba 2025-11-27 03:18:51,560 - INFO - Policy DISABLED on VMI 747aeae1-6287-42c5-a47e-1160c3aa7fb0 2025-11-27 03:18:51,576 - INFO - ECMP routes found in agent 10.0.0.19, for prefix 3912:6fe9:9a3d:f0a5:da40:767a::/96 2025-11-27 03:18:51,577 - INFO - Tap interface tapfc6b1bf8-8e found in ecmp routes in agent 10.0.0.19 2025-11-27 03:18:51,577 - INFO - Tap interface tap747aeae1-62 found in ecmp routes in agent 10.0.0.19 2025-11-27 03:18:51,592 - INFO - ECMP routes found in agent 10.0.0.19, for prefix 3912:6fe9:9a3d:f0a5:da40:767a::/96 2025-11-27 03:18:51,681 - DEBUG - Executing command: sudo tcpdump -nni tapfc6b1bf8-8e -U '(udp and src host 38b3:f805:cfff:3f05:a32a:4df9:0:3 and dst host 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292 and dst port 11000)' -w /tmp/tapfc6b1bf8-8e_ctest-random-04340788.pcap 2025-11-27 03:18:52,202 - DEBUG - Executing command: sudo tcpdump -nni tap747aeae1-62 -U '(udp and src host 38b3:f805:cfff:3f05:a32a:4df9:0:3 and dst host 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292 and dst port 11000)' -w /tmp/tap747aeae1-62_ctest-random-87404489.pcap 2025-11-27 03:18:52,289 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=icmp_error;nc -6 -q 2 -u -p 21859 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292 11000 < icmp_error, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-11-27 03:18:52,290 - DEBUG - dd bs=100 count=1 if=/dev/zero of=icmp_error;nc -6 -q 2 -u -p 21859 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292 11000 < icmp_error 2025-11-27 03:19:31,964 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.000844398 s, 118 kB/s 2025-11-27 03:19:31,964 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tapfc6b1bf8-8e_ctest-random-04340788.pcap | grep -c length 2025-11-27 03:19:31,978 - DEBUG - STDOUT: 1 2025-11-27 03:19:31,978 - DEBUG - STDERR: reading from file /tmp/tapfc6b1bf8-8e_ctest-random-04340788.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-11-27 03:19:31,978 - INFO - 1 packets are found in tcpdump output as expected 2025-11-27 03:19:31,978 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-11-27 03:19:34,025 - INFO - Tcpdump verification on VM 38b3:f805:cfff:3f05:a32a:4df9:0:4 passed 2025-11-27 03:19:34,255 - DEBUG - Forward and reverse flows for match {'sip': '38b3:f805:cfff:3f05:a32a:4df9:0:3', 'dip': '3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292', 'sport': '21859', 'dport': '11000', 'proto': '17', 'vrf_id': '2'}: 0, 0 2025-11-27 03:19:34,257 - DEBUG - Flow verification PASSED on node: 10.0.0.19 for VMs - Sender: 38b3:f805:cfff:3f05:a32a:4df9:0:3, Receiver: 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292, Flow count expected: 0 0, got: 0 0 2025-11-27 03:19:34,257 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-11-27 03:19:36,265 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-11-27 03:19:38,564 - DEBUG - Forward and reverse flows for match {'sip': '38b3:f805:cfff:3f05:a32a:4df9:0:3', 'dip': '3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292', 'sport': '21859', 'dport': '11000', 'proto': '17', 'vrf_id': '2'}: 0, 0 2025-11-27 03:19:38,565 - DEBUG - Flow verification PASSED on node: 10.0.0.130 for VMs - Sender: 38b3:f805:cfff:3f05:a32a:4df9:0:3, Receiver: 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292, Flow count expected: 0 0, got: 0 0 2025-11-27 03:19:38,565 - INFO - Traffic verification for ECMP passed 2025-11-27 03:19:38,565 - INFO - Waiting for VM ctest-DisablePolicyEcmpIpv6-67056033-52454360 to be up.. 2025-11-27 03:19:38,645 - DEBUG - VM is in ACTIVE state now 2025-11-27 03:19:38,645 - INFO - VM name : ctest-DisablePolicyEcmpIpv6-67056033-52454360 2025-11-27 03:19:38,732 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-52454360 ID is 858c56f6-a09e-4603-9e5e-969e964a4b37 2025-11-27 03:19:38,733 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-52454360 launched on Node an-jenkins-deploy-platform-ansible-os-4820-2 2025-11-27 03:19:38,821 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine/858c56f6-a09e-4603-9e5e-969e964a4b37 2025-11-27 03:19:38,830 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-machine-interface/e03dccc0-90c6-4ba8-9728-eded971a0e18 2025-11-27 03:19:39,984 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.39 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.375 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.375/1.882/3.390/1.507 ms') 2025-11-27 03:19:39,984 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-DisablePolicyEcmpIpv6-67056033-52454360 passed 2025-11-27 03:19:40,054 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-27 03:19:40,054 - DEBUG - Waiting to SSH to VM ctest-DisablePolicyEcmpIpv6-67056033-52454360, IP 38b3:f805:cfff:3f05:a32a:4df9:0:6, Port 22 2025-11-27 03:19:40,222 - DEBUG - VM ctest-DisablePolicyEcmpIpv6-67056033-52454360 is ready for SSH connections 2025-11-27 03:19:40,288 - INFO - Policy DISABLED on VMI e03dccc0-90c6-4ba8-9728-eded971a0e18 2025-11-27 03:19:40,288 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-11-27 03:19:40,288 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep 38b3:f805:cfff:3f05:a32a:4df9:0:6 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-11-27 03:19:40,288 - DEBUG - ifconfig -a| grep 38b3:f805:cfff:3f05:a32a:4df9:0:6 -A2 -B4| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1 2025-11-27 03:20:19,374 - DEBUG - eth0 2025-11-27 03:20:19,374 - DEBUG - Running remote_cmd, Cmd : sysctl net.ipv6.conf.eth0.accept_dad=0, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-11-27 03:20:19,374 - DEBUG - sysctl net.ipv6.conf.eth0.accept_dad=0 2025-11-27 03:20:19,923 - DEBUG - net.ipv6.conf.eth0.accept_dad = 0 2025-11-27 03:20:20,140 - INFO - Created InterfaceRouteTable ['default-domain', 'ctest-DisablePolicyEcmpIpv6-67056033', 'ctest-intf-rtb-26286445'](UUID 8ae429b3-a9b5-47a3-86ae-bdcaafdadb9b), prefixes : ['3912:6fe9:9a3d:f0a5:da40:767a::/96'] 2025-11-27 03:20:20,204 - INFO - Added intf route table 8ae429b3-a9b5-47a3-86ae-bdcaafdadb9b to port e03dccc0-90c6-4ba8-9728-eded971a0e18 2025-11-27 03:20:20,204 - DEBUG - Running remote_cmd, Cmd : ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-11-27 03:20:20,204 - DEBUG - ifconfig -a| grep -i 'hwaddr\|flags' | awk '{print $1}' | cut -d ':' -f 1 2025-11-27 03:20:20,668 - DEBUG - eth0 2025-11-27 03:20:20,668 - DEBUG - Running remote_cmd, Cmd : ifconfig eth0 inet6 add 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-11-27 03:20:20,668 - DEBUG - ifconfig eth0 inet6 add 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292 2025-11-27 03:20:21,171 - DEBUG - None 2025-11-27 03:20:21,172 - DEBUG - Running remote_cmd, Cmd : ifconfig -a, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-11-27 03:20:21,172 - DEBUG - ifconfig -a 2025-11-27 03:20:21,680 - DEBUG - eth0 Link encap:Ethernet HWaddr 02:e0:3d:cc:c0:90 inet addr:68.78.88.198 Bcast:68.78.88.255 Mask:255.255.255.192 inet6 addr: 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292/128 Scope:Global inet6 addr: 38b3:f805:cfff:3f05:a32a:4df9:0:6/64 Scope:Global inet6 addr: fe80::e0:3dff:fecc:c090/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:555 errors:0 dropped:0 overruns:0 frame:0 TX packets:499 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:50635 (50.6 KB) TX bytes:43196 (43.1 KB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) 2025-11-27 03:20:21,680 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-67056033-63678874 2025-11-27 03:20:21,816 - INFO - Removed intf route table 4df947a3-2d57-4643-95ef-3bbd8bd244cd from port fc6b1bf8-8e6d-4d2d-9d9f-e5af401628ba 2025-11-27 03:20:21,853 - INFO - Deleted Interface route table 4df947a3-2d57-4643-95ef-3bbd8bd244cd 2025-11-27 03:20:21,875 - INFO - ECMP routes found in agent 10.0.0.19, for prefix 3912:6fe9:9a3d:f0a5:da40:767a::/96 2025-11-27 03:20:21,875 - INFO - Tap interface tap747aeae1-62 found in ecmp routes in agent 10.0.0.19 2025-11-27 03:20:21,888 - INFO - ECMP routes found in agent 10.0.0.130, for prefix 3912:6fe9:9a3d:f0a5:da40:767a::/96 2025-11-27 03:20:21,888 - INFO - Tap interface tape03dccc0-90 found in ecmp routes in agent 10.0.0.130 2025-11-27 03:20:21,978 - DEBUG - Executing command: sudo tcpdump -nni tap747aeae1-62 -U '(udp and src host 38b3:f805:cfff:3f05:a32a:4df9:0:3 and dst host 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292 and dst port 11000)' -w /tmp/tap747aeae1-62_ctest-random-30874163.pcap 2025-11-27 03:20:22,477 - DEBUG - Executing command: sudo tcpdump -nni tape03dccc0-90 -U '(udp and src host 38b3:f805:cfff:3f05:a32a:4df9:0:3 and dst host 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292 and dst port 11000)' -w /tmp/tape03dccc0-90_ctest-random-38521230.pcap 2025-11-27 03:20:22,833 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=icmp_error;nc -6 -q 2 -u -p 37409 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292 11000 < icmp_error, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-11-27 03:20:22,834 - DEBUG - dd bs=100 count=1 if=/dev/zero of=icmp_error;nc -6 -q 2 -u -p 37409 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292 11000 < icmp_error 2025-11-27 03:20:25,366 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.00156092 s, 64.1 kB/s 2025-11-27 03:20:25,366 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/tap747aeae1-62_ctest-random-30874163.pcap | grep -c length 2025-11-27 03:20:25,381 - DEBUG - STDOUT: 1 2025-11-27 03:20:25,381 - DEBUG - STDERR: reading from file /tmp/tap747aeae1-62_ctest-random-30874163.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-11-27 03:20:25,381 - INFO - 1 packets are found in tcpdump output as expected 2025-11-27 03:20:25,381 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-11-27 03:20:27,425 - INFO - Tcpdump verification on VM 38b3:f805:cfff:3f05:a32a:4df9:0:5 passed 2025-11-27 03:20:27,660 - DEBUG - Forward and reverse flows for match {'sip': '38b3:f805:cfff:3f05:a32a:4df9:0:3', 'dip': '3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292', 'sport': '37409', 'dport': '11000', 'proto': '17', 'vrf_id': '2'}: 0, 0 2025-11-27 03:20:27,661 - DEBUG - Flow verification PASSED on node: 10.0.0.19 for VMs - Sender: 38b3:f805:cfff:3f05:a32a:4df9:0:3, Receiver: 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292, Flow count expected: 0 0, got: 0 0 2025-11-27 03:20:27,661 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-11-27 03:20:29,670 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-11-27 03:20:31,950 - DEBUG - Forward and reverse flows for match {'sip': '38b3:f805:cfff:3f05:a32a:4df9:0:3', 'dip': '3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292', 'sport': '37409', 'dport': '11000', 'proto': '17', 'vrf_id': '2'}: 0, 0 2025-11-27 03:20:31,950 - DEBUG - Flow verification PASSED on node: 10.0.0.130 for VMs - Sender: 38b3:f805:cfff:3f05:a32a:4df9:0:3, Receiver: 3912:6fe9:9a3d:f0a5:da40:767a:bee3:8292, Flow count expected: 0 0, got: 0 0 2025-11-27 03:20:31,950 - INFO - Traffic verification for ECMP passed 2025-11-27 03:20:31,950 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-67056033-52454360 2025-11-27 03:20:32,084 - INFO - Removed intf route table 8ae429b3-a9b5-47a3-86ae-bdcaafdadb9b from port e03dccc0-90c6-4ba8-9728-eded971a0e18 2025-11-27 03:20:32,145 - INFO - Deleted Interface route table 8ae429b3-a9b5-47a3-86ae-bdcaafdadb9b 2025-11-27 03:20:32,145 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-67056033-10758646 2025-11-27 03:20:32,292 - INFO - Removed intf route table ac2f06b2-af30-4575-870a-2b3a4ed37051 from port 747aeae1-6287-42c5-a47e-1160c3aa7fb0 2025-11-27 03:20:32,333 - INFO - Deleted Interface route table ac2f06b2-af30-4575-870a-2b3a4ed37051 2025-11-27 03:20:32,333 - INFO - Deleting VM ctest-DisablePolicyEcmpIpv6-67056033-87001045 2025-11-27 03:20:32,433 - INFO - Deleting VN ctest-vn-43940798 2025-11-27 03:20:32,480 - DEBUG - VN 7261bb8c-8840-4bf9-a564-b655c5f649d0 still in use: Unable to complete operation on network 7261bb8c-8840-4bf9-a564-b655c5f649d0. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-ae74c277-d628-4416-ac18-375ef8acfd28'] 2025-11-27 03:20:32,481 - WARNING - Deleting VN ctest-vn-43940798 failed..Will retry 2025-11-27 03:20:34,694 - DEBUG - Response for deleting network () 2025-11-27 03:20:34,694 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/eb87cc2f-0c7a-4931-94b1-dab0e2deb176 2025-11-27 03:20:34,704 - DEBUG - Response Code: 404 2025-11-27 03:20:34,704 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-11-27 03:20:34,711 - DEBUG - Requesting: http://10.0.0.27:8082/domain/ea95f9c6-3b7f-4f10-9ecb-80826711f59a 2025-11-27 03:20:34,789 - DEBUG - Requesting: http://10.0.0.27:8082/project/633eec2a-4e6c-43e6-af7e-507d29215bf6 2025-11-27 03:20:34,882 - INFO - Validated that VN ctest-vn-43940798 is not found in API Server 2025-11-27 03:20:34,900 - DEBUG - VN ctest-vn-43940798 is not present in Agent 10.0.0.19 2025-11-27 03:20:34,911 - DEBUG - VN ctest-vn-43940798 is not present in Agent 10.0.0.130 2025-11-27 03:20:34,911 - INFO - Validated that VN ctest-vn-43940798 is not in any agent 2025-11-27 03:20:34,945 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn-43940798 info 2025-11-27 03:20:36,223 - DEBUG - No XMPP flaps were noticed during the test
2025-11-27 03:20:36,223 - INFO - END TEST : test_ecmp_with_static_routes : PASSED[0:04:17]
2025-11-27 03:20:36,223 - INFO - -------------------------------------------------------------------------------- 2025-11-27 03:20:37,010 - INFO - Deleted project: ctest-DisablePolicyEcmpIpv6-67056033, ID : 633eec2a-4e6c-43e6-af7e-507d29215bf6