2025-07-23 17:07:01,774 - INFO - Domain Default found not creating 2025-07-23 17:07:01,924 - INFO - Project ctest-TestMtuProbe-55648472 not found, creating it 2025-07-23 17:07:02,489 - INFO - Created Project:ctest-TestMtuProbe-55648472, ID : 1d70cd1f-ffcb-4d19-b486-a265a2179d49 2025-07-23 17:07:04,024 - INFO - ================================================================================ 2025-07-23 17:07:04,024 - INFO - STARTING TEST : test_mtu_probe_inter_node 2025-07-23 17:07:04,025 - INFO - TEST DESCRIPTION : Description: check traffic configuring different tcp_mtu_probe vlaues on vm MTU probing values: 0, 1, 2 0 - Disabled (default) 1 - Disabled by default, Enabled only when ICMP black hole detected 2 - Always enabled, using the initial MSS value of tcp_base_mss system internally enables bit values for point 1&2 For test purpose, Idea is to check traffic works fine on configuring values Vm's across compute node steps: 1. create 1 VN and launch 2 VMs across compute node 2. client and server VMs across compute node 3. configure tcp_mtu_probe to 0, 1, 2 4. send icmp traffic assert if ping fails 5. send tcp traffic assert if tcp traffic fails Pass criteria: 1. If icmp/tcp traffic work fine configuring tcp_mtu_probe values 2025-07-23 17:07:05,383 - DEBUG - Nothing to compare xmpp stats {'10.0.0.50': {'10.20.0.254': '0', '10.20.0.17': '0'}, '10.0.0.130': {'10.20.0.17': '0', '10.20.0.254': '0'}} with 2025-07-23 17:07:05,384 - INFO - Initial checks done. Running the testcase now 2025-07-23 17:07:05,384 - INFO - 2025-07-23 17:07:05,407 - DEBUG - Services list from nova: [, , , ] 2025-07-23 17:07:05,407 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3704-2', 'an-jenkins-deploy-platform-ansible-os-3704-1']} 2025-07-23 17:07:06,128 - DEBUG - Response for create_network : {'network': {'id': '9ad350c9-fb80-4d1b-b864-98317d67d591', 'name': 'ctest-vn-26558574', 'tenant_id': '1d70cd1fffcb4d19b486a265a2179d49', 'project_id': '1d70cd1fffcb4d19b486a265a2179d49', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestMtuProbe-55648472', 'ctest-vn-26558574'], 'port_security_enabled': True, 'description': ''}} 2025-07-23 17:07:06,458 - DEBUG - Response for create_subnet : {'subnet': {'id': '67e12e7f-fae3-4f46-9f5e-ee7bd1ace4ae', 'name': '', 'tenant_id': '1d70cd1fffcb4d19b486a265a2179d49', 'network_id': '9ad350c9-fb80-4d1b-b864-98317d67d591', 'ip_version': 4, 'cidr': '109.17.209.192/26', 'allocation_pools': [{'start': '109.17.209.194', 'end': '109.17.209.254'}], 'gateway_ip': '109.17.209.193', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '109.17.209.194', 'tags': [], 'project_id': '1d70cd1fffcb4d19b486a265a2179d49'}} 2025-07-23 17:07:06,485 - INFO - Created VN ctest-vn-26558574 2025-07-23 17:07:06,498 - DEBUG - VN ctest-vn-26558574 UUID is 9ad350c9-fb80-4d1b-b864-98317d67d591 2025-07-23 17:07:06,580 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-07-23 17:07:06,913 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-07-23 17:07:06,943 - DEBUG - Requesting: http://10.0.0.23:8082/domain/85afe8be-97e7-4e27-9667-ee7272dde692 2025-07-23 17:07:06,963 - DEBUG - Requesting: http://10.0.0.23:8082/project/1d70cd1f-ffcb-4d19-b486-a265a2179d49 2025-07-23 17:07:07,072 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/9ad350c9-fb80-4d1b-b864-98317d67d591 2025-07-23 17:07:07,085 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/9ad350c9-fb80-4d1b-b864-98317d67d591 2025-07-23 17:07:07,095 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/8e6c45af-5d2a-4ddd-a3b9-ba853313210f 2025-07-23 17:07:07,106 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/8e6c45af-5d2a-4ddd-a3b9-ba853313210f 2025-07-23 17:07:07,115 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/b734bbb3-a9d3-4739-8836-9cebff23e03e 2025-07-23 17:07:07,123 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-23 17:07:07,123 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/9ad350c9-fb80-4d1b-b864-98317d67d591 2025-07-23 17:07:07,135 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/8e6c45af-5d2a-4ddd-a3b9-ba853313210f 2025-07-23 17:07:07,155 - INFO - Verified VN network id 12 for VN 9ad350c9-fb80-4d1b-b864-98317d67d591 2025-07-23 17:07:07,155 - INFO - Verifications in API Server for VN ctest-vn-26558574 passed 2025-07-23 17:07:07,155 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/9ad350c9-fb80-4d1b-b864-98317d67d591 2025-07-23 17:07:07,168 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/8e6c45af-5d2a-4ddd-a3b9-ba853313210f 2025-07-23 17:07:07,176 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/8e6c45af-5d2a-4ddd-a3b9-ba853313210f 2025-07-23 17:07:07,185 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/b734bbb3-a9d3-4739-8836-9cebff23e03e 2025-07-23 17:07:07,203 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-55648472', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '11156349530364988699', 'uuid-lslong': '13286912138974582161'}, 'enable': 'true', 'created': '2025-07-23T17:07:06', 'last-modified': '2025-07-23T17:07:06', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.802627'} 2025-07-23 17:07:07,212 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-23 17:07:07,220 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-55648472', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '11156349530364988699', 'uuid-lslong': '13286912138974582161'}, 'enable': 'true', 'created': '2025-07-23T17:07:06', 'last-modified': '2025-07-23T17:07:06', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.826667'} 2025-07-23 17:07:07,227 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-23 17:07:07,233 - DEBUG - Control-node 10.0.0.73 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-55648472', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '11156349530364988699', 'uuid-lslong': '13286912138974582161'}, 'enable': 'true', 'created': '2025-07-23T17:07:06', 'last-modified': '2025-07-23T17:07:06', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.836247'} 2025-07-23 17:07:07,240 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-23 17:07:07,240 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn-26558574 passed 2025-07-23 17:07:07,240 - DEBUG - ====Verifying policy data for ctest-vn-26558574 in API_Server ====== 2025-07-23 17:07:07,240 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-07-23 17:07:07,249 - DEBUG - Requesting: http://10.0.0.23:8082/domain/85afe8be-97e7-4e27-9667-ee7272dde692 2025-07-23 17:07:07,267 - DEBUG - Requesting: http://10.0.0.23:8082/project/1d70cd1f-ffcb-4d19-b486-a265a2179d49 2025-07-23 17:07:07,372 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/9ad350c9-fb80-4d1b-b864-98317d67d591 2025-07-23 17:07:07,385 - DEBUG - =>VN ctest-vn-26558574 has no policy to be verified 2025-07-23 17:07:07,385 - DEBUG - Verifying the vn in opserver 2025-07-23 17:07:07,386 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574 virtual network link through opserver 10.0.0.23 2025-07-23 17:07:07,386 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-07-23 17:07:07,689 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-07-23 17:07:07,762 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574?flat'} 2025-07-23 17:07:07,762 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574 virtual network link through opserver 10.0.0.249 2025-07-23 17:07:07,763 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2025-07-23 17:07:08,058 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2025-07-23 17:07:08,123 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574?flat'} 2025-07-23 17:07:08,123 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574 virtual network link through opserver 10.0.0.73 2025-07-23 17:07:08,123 - DEBUG - Requesting: http://10.0.0.73:8081/analytics/uves/virtual-networks 2025-07-23 17:07:08,425 - DEBUG - Requesting: http://10.0.0.73:8081/analytics/uves/virtual-networks 2025-07-23 17:07:08,487 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574', 'href': 'http://10.0.0.73:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574?flat'} 2025-07-23 17:07:08,487 - INFO - Validated that VN default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574 is found in opserver 2025-07-23 17:07:08,532 - DEBUG - Do not have enough data to verify VN in agent 2025-07-23 17:07:08,553 - DEBUG - VRF ids for VN ctest-vn-26558574: {} 2025-07-23 17:07:08,553 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3704-2', 'an-jenkins-deploy-platform-ansible-os-3704-1']} 2025-07-23 17:07:08,553 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3704-2', 'an-jenkins-deploy-platform-ansible-os-3704-1']} 2025-07-23 17:07:09,715 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3704-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3704-2) 2025-07-23 17:07:09,715 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3704-2', 'an-jenkins-deploy-platform-ansible-os-3704-1']} 2025-07-23 17:07:11,231 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3704-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3704-1) 2025-07-23 17:07:11,231 - INFO - Waiting for VM ctest-TestMtuProbe-55648472-99255584 to be up.. 2025-07-23 17:07:11,326 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-23 17:07:16,404 - DEBUG - VM is in ACTIVE state now 2025-07-23 17:07:16,405 - INFO - VM name : ctest-TestMtuProbe-55648472-99255584 2025-07-23 17:07:16,482 - DEBUG - VM ctest-TestMtuProbe-55648472-99255584 ID is c6f57dc8-5bcf-4381-a191-290172dc7a21 2025-07-23 17:07:16,506 - DEBUG - VM ctest-TestMtuProbe-55648472-99255584 launched on Node an-jenkins-deploy-platform-ansible-os-3704-2 2025-07-23 17:07:16,599 - DEBUG - Requesting: http://10.0.0.73:8082/virtual-machine/c6f57dc8-5bcf-4381-a191-290172dc7a21 2025-07-23 17:07:16,901 - DEBUG - Requesting: http://10.0.0.73:8082/virtual-machine/c6f57dc8-5bcf-4381-a191-290172dc7a21 2025-07-23 17:07:16,941 - DEBUG - Requesting: http://10.0.0.73:8082/virtual-machine-interface/3da0680c-f321-4ca7-a034-5406bf924e57 2025-07-23 17:07:20,200 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1029ms') 2025-07-23 17:07:20,200 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-99255584 failed! 2025-07-23 17:07:20,217 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574 is 109.17.209.193 and allocation pool is NOT set 2025-07-23 17:07:24,294 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-07-23 17:07:24,295 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-99255584 failed! 2025-07-23 17:07:24,316 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574 is 109.17.209.193 and allocation pool is NOT set 2025-07-23 17:07:28,388 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms') 2025-07-23 17:07:28,389 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-99255584 failed! 2025-07-23 17:07:28,407 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574 is 109.17.209.193 and allocation pool is NOT set 2025-07-23 17:07:32,486 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1007ms') 2025-07-23 17:07:32,486 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-99255584 failed! 2025-07-23 17:07:32,501 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574 is 109.17.209.193 and allocation pool is NOT set 2025-07-23 17:07:36,582 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-07-23 17:07:36,582 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-99255584 failed! 2025-07-23 17:07:36,597 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574 is 109.17.209.193 and allocation pool is NOT set 2025-07-23 17:07:40,679 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-07-23 17:07:40,680 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-99255584 failed! 2025-07-23 17:07:40,697 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574 is 109.17.209.193 and allocation pool is NOT set 2025-07-23 17:07:44,774 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-07-23 17:07:44,775 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-99255584 failed! 2025-07-23 17:07:44,790 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574 is 109.17.209.193 and allocation pool is NOT set 2025-07-23 17:07:48,864 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-07-23 17:07:48,864 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-99255584 failed! 2025-07-23 17:07:48,881 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574 is 109.17.209.193 and allocation pool is NOT set 2025-07-23 17:07:52,960 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-07-23 17:07:52,960 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-99255584 failed! 2025-07-23 17:07:52,976 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-26558574 is 109.17.209.193 and allocation pool is NOT set 2025-07-23 17:07:55,048 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=8.48 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=2.89 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 2.892/5.685/8.478/2.793 ms') 2025-07-23 17:07:55,048 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-99255584 passed 2025-07-23 17:07:55,164 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:07:55,164 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-55648472-99255584, IP 109.17.209.195, Port 22 2025-07-23 17:07:55,232 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-23 17:07:55,322 - DEBUG - VM ctest-TestMtuProbe-55648472-99255584 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-23 17:08:00,323 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:08:00,323 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-55648472-99255584, IP 109.17.209.195, Port 22 2025-07-23 17:08:00,381 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-23 17:08:00,471 - DEBUG - VM ctest-TestMtuProbe-55648472-99255584 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-23 17:08:05,472 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:08:05,473 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-55648472-99255584, IP 109.17.209.195, Port 22 2025-07-23 17:08:05,539 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-23 17:08:05,623 - DEBUG - VM ctest-TestMtuProbe-55648472-99255584 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-23 17:08:10,623 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:08:10,623 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-55648472-99255584, IP 109.17.209.195, Port 22 2025-07-23 17:08:10,681 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-23 17:08:10,768 - DEBUG - VM ctest-TestMtuProbe-55648472-99255584 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-23 17:08:15,769 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:08:15,770 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-55648472-99255584, IP 109.17.209.195, Port 22 2025-07-23 17:08:15,933 - DEBUG - VM ctest-TestMtuProbe-55648472-99255584 is ready for SSH connections 2025-07-23 17:08:15,934 - INFO - Waiting for VM ctest-TestMtuProbe-55648472-01820177 to be up.. 2025-07-23 17:08:16,024 - DEBUG - VM is in ACTIVE state now 2025-07-23 17:08:16,025 - INFO - VM name : ctest-TestMtuProbe-55648472-01820177 2025-07-23 17:08:16,110 - DEBUG - VM ctest-TestMtuProbe-55648472-01820177 ID is 4164494e-5ed8-488b-83ca-aed974b34d4d 2025-07-23 17:08:16,110 - DEBUG - VM ctest-TestMtuProbe-55648472-01820177 launched on Node an-jenkins-deploy-platform-ansible-os-3704-1 2025-07-23 17:08:16,186 - DEBUG - Requesting: http://10.0.0.73:8082/virtual-machine/4164494e-5ed8-488b-83ca-aed974b34d4d 2025-07-23 17:08:16,196 - DEBUG - Requesting: http://10.0.0.73:8082/virtual-machine-interface/7e87a67d-3edc-4f81-824e-68c5c6c0686b 2025-07-23 17:08:17,443 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=8.48 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=2.74 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 2.739/5.610/8.482/2.871 ms') 2025-07-23 17:08:17,444 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-01820177 passed 2025-07-23 17:08:17,562 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:08:17,563 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-55648472-01820177, IP 109.17.209.196, Port 22 2025-07-23 17:08:17,756 - DEBUG - VM ctest-TestMtuProbe-55648472-01820177 is ready for SSH connections 2025-07-23 17:08:17,756 - DEBUG - Running remote_cmd, Cmd : echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:08:17,757 - DEBUG - echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-23 17:08:35,270 - DEBUG - None 2025-07-23 17:08:35,270 - DEBUG - Running remote_cmd, Cmd : echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:08:35,270 - DEBUG - echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-23 17:09:14,920 - DEBUG - None 2025-07-23 17:09:14,920 - INFO - {'echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-23 17:09:14,921 - INFO - {'echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-23 17:09:14,922 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:09:14,922 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 109.17.209.196, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:09:14,922 - DEBUG - ping -s 56 -c 3 -W 1 109.17.209.196 2025-07-23 17:09:17,394 - DEBUG - PING 109.17.209.196 (109.17.209.196) 56(84) bytes of data. 64 bytes from 109.17.209.196: icmp_req=1 ttl=64 time=18.3 ms 64 bytes from 109.17.209.196: icmp_req=2 ttl=64 time=1.60 ms 64 bytes from 109.17.209.196: icmp_req=3 ttl=64 time=1.72 ms --- 109.17.209.196 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.609/7.238/18.381/7.879 ms 2025-07-23 17:09:17,394 - INFO - Ping to IP 109.17.209.196 from VM ctest-TestMtuProbe-55648472-99255584 passed 2025-07-23 17:09:17,394 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:09:17,394 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:09:17,394 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile & 2025-07-23 17:09:18,054 - DEBUG - total 132 drwxr-xr-x 11 ubuntu ubuntu 4096 Dec 29 2017 . drwxr-xr-x 3 root root 4096 Jan 4 2013 .. -rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history -rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout -rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache -rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins -rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla -rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh -rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh -rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0 -rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz -rw------- 1 root root 4695 Dec 17 2017 .viminfo -rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority -rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors 2025-07-23 17:09:18,054 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:09:18,054 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile & 2025-07-23 17:09:18,496 - DEBUG - None 2025-07-23 17:09:18,496 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 109.17.209.196 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:09:18,497 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 109.17.209.196 5201 < testfile 2025-07-23 17:09:24,022 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.000738428 s, 135 kB/s 2025-07-23 17:09:24,022 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:09:24,022 - DEBUG - ls -l testfile 2025-07-23 17:09:24,421 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:09:24,421 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:09:27,421 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:09:27,422 - DEBUG - ls -l testfile 2025-07-23 17:09:27,824 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:09:27,824 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:09:30,825 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:09:30,825 - DEBUG - ls -l testfile 2025-07-23 17:09:31,251 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:09:31,251 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:09:34,251 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:09:34,251 - DEBUG - ls -l testfile 2025-07-23 17:09:34,654 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:09:34,654 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:09:37,655 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:09:37,655 - DEBUG - ls -l testfile 2025-07-23 17:09:38,075 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:09:38,076 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:09:41,076 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:09:41,077 - DEBUG - ls -l testfile 2025-07-23 17:09:41,480 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:09:41,481 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:09:44,481 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:09:44,481 - DEBUG - ls -l testfile 2025-07-23 17:09:44,840 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:09:44,841 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:09:47,842 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:09:47,842 - DEBUG - ls -l testfile 2025-07-23 17:09:48,226 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:09:48,227 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:09:51,227 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:09:51,228 - DEBUG - ls -l testfile 2025-07-23 17:09:51,629 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:09:51,629 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:09:54,630 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:09:54,630 - DEBUG - ls -l testfile 2025-07-23 17:09:55,015 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:09:55,015 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:09:58,016 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:09:58,016 - DEBUG - ls -l testfile 2025-07-23 17:09:58,426 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:09:58,426 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:09:58,426 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-55648472-01820177 2025-07-23 17:09:58,426 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 109.17.209.196 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:09:58,426 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 109.17.209.196 5201 < testfile 2025-07-23 17:10:03,993 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.000940046 s, 106 kB/s 2025-07-23 17:10:03,993 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:03,993 - DEBUG - ls -l testfile 2025-07-23 17:10:04,406 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:04,406 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:07,407 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:07,408 - DEBUG - ls -l testfile 2025-07-23 17:10:07,830 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:07,830 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:10,830 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:10,831 - DEBUG - ls -l testfile 2025-07-23 17:10:11,185 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:11,185 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:14,186 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:14,186 - DEBUG - ls -l testfile 2025-07-23 17:10:14,514 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:14,515 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:17,515 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:17,515 - DEBUG - ls -l testfile 2025-07-23 17:10:17,870 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:17,870 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:20,871 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:20,871 - DEBUG - ls -l testfile 2025-07-23 17:10:21,337 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:21,337 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:24,338 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:24,338 - DEBUG - ls -l testfile 2025-07-23 17:10:24,745 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:24,745 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:27,746 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:27,746 - DEBUG - ls -l testfile 2025-07-23 17:10:28,162 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:28,162 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:31,163 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:31,163 - DEBUG - ls -l testfile 2025-07-23 17:10:31,959 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:31,959 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:34,960 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:34,960 - DEBUG - ls -l testfile 2025-07-23 17:10:35,365 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:35,365 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:38,366 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:38,366 - DEBUG - ls -l testfile 2025-07-23 17:10:38,773 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:38,774 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:38,774 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-55648472-01820177 2025-07-23 17:10:38,774 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 109.17.209.196 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:10:38,774 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 109.17.209.196 5201 < testfile 2025-07-23 17:10:44,319 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.000940206 s, 106 kB/s 2025-07-23 17:10:44,320 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:44,320 - DEBUG - ls -l testfile 2025-07-23 17:10:44,668 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:44,668 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:47,668 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:47,669 - DEBUG - ls -l testfile 2025-07-23 17:10:48,106 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:48,107 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:51,108 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:51,108 - DEBUG - ls -l testfile 2025-07-23 17:10:51,476 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:51,476 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:54,477 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:54,477 - DEBUG - ls -l testfile 2025-07-23 17:10:54,869 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:54,869 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:10:57,870 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:10:57,870 - DEBUG - ls -l testfile 2025-07-23 17:10:58,279 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:10:58,279 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:01,279 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:01,280 - DEBUG - ls -l testfile 2025-07-23 17:11:01,668 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:01,668 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:04,669 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:04,669 - DEBUG - ls -l testfile 2025-07-23 17:11:05,052 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:05,052 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:08,053 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:08,053 - DEBUG - ls -l testfile 2025-07-23 17:11:08,394 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:08,394 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:11,395 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:11,395 - DEBUG - ls -l testfile 2025-07-23 17:11:11,780 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:11,780 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:14,781 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:14,781 - DEBUG - ls -l testfile 2025-07-23 17:11:15,165 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:15,165 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:18,165 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:18,166 - DEBUG - ls -l testfile 2025-07-23 17:11:18,562 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:18,562 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:18,562 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-55648472-01820177 2025-07-23 17:11:18,562 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 109.17.209.196 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:11:18,563 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 109.17.209.196 5201 < testfile 2025-07-23 17:11:24,104 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.000885005 s, 113 kB/s 2025-07-23 17:11:24,104 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:24,104 - DEBUG - ls -l testfile 2025-07-23 17:11:24,498 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:24,498 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:27,499 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:27,499 - DEBUG - ls -l testfile 2025-07-23 17:11:27,888 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:27,888 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:30,889 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:30,889 - DEBUG - ls -l testfile 2025-07-23 17:11:31,287 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:31,288 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:34,288 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:34,288 - DEBUG - ls -l testfile 2025-07-23 17:11:34,692 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:34,693 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:37,693 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:37,693 - DEBUG - ls -l testfile 2025-07-23 17:11:38,132 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:38,132 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:41,132 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:41,133 - DEBUG - ls -l testfile 2025-07-23 17:11:41,521 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:41,521 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:44,521 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:44,522 - DEBUG - ls -l testfile 2025-07-23 17:11:44,942 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:44,942 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:47,943 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:47,943 - DEBUG - ls -l testfile 2025-07-23 17:11:48,367 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:48,367 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:51,367 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:51,368 - DEBUG - ls -l testfile 2025-07-23 17:11:51,723 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:51,723 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:54,725 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:54,725 - DEBUG - ls -l testfile 2025-07-23 17:11:55,099 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:55,099 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:58,100 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:11:58,101 - DEBUG - ls -l testfile 2025-07-23 17:11:58,490 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:11:58,490 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:11:58,490 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-55648472-01820177 2025-07-23 17:11:58,490 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 109.17.209.196 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:11:58,490 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 109.17.209.196 5201 < testfile 2025-07-23 17:12:04,041 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.000934226 s, 107 kB/s 2025-07-23 17:12:04,041 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:12:04,041 - DEBUG - ls -l testfile 2025-07-23 17:12:04,438 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:12:04,438 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:12:07,439 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:12:07,439 - DEBUG - ls -l testfile 2025-07-23 17:12:07,754 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:12:07,755 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:12:10,755 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:12:10,755 - DEBUG - ls -l testfile 2025-07-23 17:12:11,183 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:12:11,183 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:12:14,184 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:12:14,184 - DEBUG - ls -l testfile 2025-07-23 17:12:14,564 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:12:14,564 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:12:17,565 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:12:17,565 - DEBUG - ls -l testfile 2025-07-23 17:12:17,956 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:12:17,956 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:12:20,957 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:12:20,957 - DEBUG - ls -l testfile 2025-07-23 17:12:21,351 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:12:21,352 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:12:24,352 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:12:24,352 - DEBUG - ls -l testfile 2025-07-23 17:12:24,741 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:12:24,741 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:12:27,742 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:12:27,742 - DEBUG - ls -l testfile 2025-07-23 17:12:28,140 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:12:28,140 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:12:31,141 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:12:31,141 - DEBUG - ls -l testfile 2025-07-23 17:12:31,578 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:12:31,578 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:12:34,579 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:12:34,579 - DEBUG - ls -l testfile 2025-07-23 17:12:34,970 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:12:34,970 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:12:37,970 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.50, gateway password: c0ntrail123 2025-07-23 17:12:37,970 - DEBUG - ls -l testfile 2025-07-23 17:12:38,365 - DEBUG - -rw-r--r-- 1 root root 0 Jul 23 17:09 testfile 2025-07-23 17:12:38,365 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds 2025-07-23 17:12:38,365 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-55648472-01820177 2025-07-23 17:12:38,367 - INFO - Deleting VM ctest-TestMtuProbe-55648472-01820177 2025-07-23 17:12:38,468 - INFO - Deleting VM ctest-TestMtuProbe-55648472-99255584 2025-07-23 17:12:38,538 - INFO - Deleting VN ctest-vn-26558574 2025-07-23 17:12:38,630 - DEBUG - VN 9ad350c9-fb80-4d1b-b864-98317d67d591 still in use: Unable to complete operation on network 9ad350c9-fb80-4d1b-b864-98317d67d591. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-be1d5d49-752c-467e-bd1e-cc8ea993a5e0'] 2025-07-23 17:12:38,630 - WARNING - Deleting VN ctest-vn-26558574 failed..Will retry 2025-07-23 17:12:40,867 - DEBUG - Response for deleting network () 2025-07-23 17:12:40,867 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/8e6c45af-5d2a-4ddd-a3b9-ba853313210f 2025-07-23 17:12:40,912 - DEBUG - Response Code: 404 2025-07-23 17:12:40,912 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-07-23 17:12:40,919 - DEBUG - Requesting: http://10.0.0.23:8082/domain/85afe8be-97e7-4e27-9667-ee7272dde692 2025-07-23 17:12:40,942 - DEBUG - Requesting: http://10.0.0.23:8082/project/1d70cd1f-ffcb-4d19-b486-a265a2179d49 2025-07-23 17:12:41,048 - INFO - Validated that VN ctest-vn-26558574 is not found in API Server 2025-07-23 17:12:41,074 - DEBUG - VN ctest-vn-26558574 is not present in Agent 10.0.0.50 2025-07-23 17:12:41,088 - DEBUG - VN ctest-vn-26558574 is not present in Agent 10.0.0.130 2025-07-23 17:12:41,088 - INFO - Validated that VN ctest-vn-26558574 is not in any agent 2025-07-23 17:12:41,144 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn-26558574 info 2025-07-23 17:12:42,497 - ERROR - AssertionError Python 3.6.8: /usr/bin/python3 Wed Jul 23 17:12:38 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/tcutils/util.py in wrapper(self=, *func_args=(), **func_kwargs={}) 1039 raise testtools.TestCase.skipException(msg) 1040 1041 return f(self, *func_args, **func_kwargs) 1042 return wrapper 1043 return decorator f = self = func_args = () func_kwargs = {} /contrail-test/scripts/vrouter/test_tcp_mtu_probe.py in test_mtu_probe_inter_node(self=) 84 self.configure_tcp_mtu_probe(client,server,value) 85 self.verify_icmp_traffic(client,server) 86 self.verify_tcp_traffic(client,server, value) 87 return True 88 self = self.verify_tcp_traffic = > client = server = value = 0 /contrail-test/scripts/vrouter/test_tcp_mtu_probe.py in verify_tcp_traffic(self=, client=, server=, value=0) 42 if value is 2: 43 sport += value 44 assert self.send_nc_traffic(client, server, sport, dport, proto), "TCP traffic failed" 45 return True 46 self = self.send_nc_traffic = > client = server = sport = 10001 dport = 5201 proto = 'tcp' AssertionError: TCP traffic failed __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('TCP traffic failed',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/tcutils/util.py", line 1041, in wrapper return f(self, *func_args, **func_kwargs) File "/contrail-test/scripts/vrouter/test_tcp_mtu_probe.py", line 86, in test_mtu_probe_inter_node self.verify_tcp_traffic(client,server, value) File "/contrail-test/scripts/vrouter/test_tcp_mtu_probe.py", line 44, in verify_tcp_traffic assert self.send_nc_traffic(client, server, sport, dport, proto), "TCP traffic failed" AssertionError: TCP traffic failed 2025-07-23 17:12:42,519 - DEBUG - No XMPP flaps were noticed during the test 2025-07-23 17:12:42,520 - INFO - 2025-07-23 17:12:42,520 - INFO - END TEST : test_mtu_probe_inter_node : FAILED[0:05:38] 2025-07-23 17:12:42,520 - INFO - -------------------------------------------------------------------------------- 2025-07-23 17:12:42,528 - INFO - ================================================================================ 2025-07-23 17:12:42,528 - INFO - STARTING TEST : test_mtu_probe_intra_node 2025-07-23 17:12:42,528 - INFO - TEST DESCRIPTION : Description: check traffic configuring different tcp_mtu_probe vlaues on vm MTU probing values: 0, 1, 2 0 - Disabled (default) 1 - Disabled by default, Enabled only when ICMP black hole detected 2 - Always enabled, using the initial MSS value of tcp_base_mss system internally enables bit values for point 1&2 For test purpose, Idea is to check traffic works fine on configuring values Vm's across compute node steps: 1. create 1 VN and launch 2 VMs in same compute node 2. client and server VMs in same compute node 3. configure tcp_mtu_probe to 0, 1, 2 4. send icmp traffic assert if ping fails 5. send tcp traffic assert if tcp traffic fails Pass criteria: 1. If icmp/tcp traffic work fine configuring tcp_mtu_probe values 2025-07-23 17:12:43,869 - DEBUG - Nothing to compare xmpp stats {'10.0.0.50': {'10.20.0.254': '0', '10.20.0.17': '0'}, '10.0.0.130': {'10.20.0.17': '0', '10.20.0.254': '0'}} with 2025-07-23 17:12:43,869 - INFO - Initial checks done. Running the testcase now 2025-07-23 17:12:43,869 - INFO - 2025-07-23 17:12:44,222 - DEBUG - Response for create_network : {'network': {'id': '72bdd76d-9342-4484-aa1a-a0ec2a01d885', 'name': 'ctest-vn-31775303', 'tenant_id': '1d70cd1fffcb4d19b486a265a2179d49', 'project_id': '1d70cd1fffcb4d19b486a265a2179d49', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestMtuProbe-55648472', 'ctest-vn-31775303'], 'port_security_enabled': True, 'description': ''}} 2025-07-23 17:12:44,520 - DEBUG - Response for create_subnet : {'subnet': {'id': '12c45a6c-f2ee-4c9d-8fc4-3f797643579e', 'name': '', 'tenant_id': '1d70cd1fffcb4d19b486a265a2179d49', 'network_id': '72bdd76d-9342-4484-aa1a-a0ec2a01d885', 'ip_version': 4, 'cidr': '59.93.191.192/26', 'allocation_pools': [{'start': '59.93.191.194', 'end': '59.93.191.254'}], 'gateway_ip': '59.93.191.193', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '59.93.191.194', 'tags': [], 'project_id': '1d70cd1fffcb4d19b486a265a2179d49'}} 2025-07-23 17:12:44,543 - INFO - Created VN ctest-vn-31775303 2025-07-23 17:12:44,561 - DEBUG - VN ctest-vn-31775303 UUID is 72bdd76d-9342-4484-aa1a-a0ec2a01d885 2025-07-23 17:12:44,646 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-07-23 17:12:44,653 - DEBUG - Requesting: http://10.0.0.23:8082/domain/85afe8be-97e7-4e27-9667-ee7272dde692 2025-07-23 17:12:44,668 - DEBUG - Requesting: http://10.0.0.23:8082/project/1d70cd1f-ffcb-4d19-b486-a265a2179d49 2025-07-23 17:12:44,764 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/72bdd76d-9342-4484-aa1a-a0ec2a01d885 2025-07-23 17:12:44,781 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/72bdd76d-9342-4484-aa1a-a0ec2a01d885 2025-07-23 17:12:44,794 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/5d6ca6be-3574-4797-9987-d53b43df0d23 2025-07-23 17:12:44,802 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/5d6ca6be-3574-4797-9987-d53b43df0d23 2025-07-23 17:12:44,810 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/d87b453e-10f2-47d7-89b1-f256e04093c3 2025-07-23 17:12:44,818 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-23 17:12:44,818 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/72bdd76d-9342-4484-aa1a-a0ec2a01d885 2025-07-23 17:12:44,827 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/5d6ca6be-3574-4797-9987-d53b43df0d23 2025-07-23 17:12:44,844 - INFO - Verified VN network id 12 for VN 72bdd76d-9342-4484-aa1a-a0ec2a01d885 2025-07-23 17:12:44,844 - INFO - Verifications in API Server for VN ctest-vn-31775303 passed 2025-07-23 17:12:44,844 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/72bdd76d-9342-4484-aa1a-a0ec2a01d885 2025-07-23 17:12:44,855 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/5d6ca6be-3574-4797-9987-d53b43df0d23 2025-07-23 17:12:44,867 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/5d6ca6be-3574-4797-9987-d53b43df0d23 2025-07-23 17:12:44,874 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/d87b453e-10f2-47d7-89b1-f256e04093c3 2025-07-23 17:12:44,886 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-55648472', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '8268001356544099460', 'uuid-lslong': '12257286272019716229'}, 'enable': 'true', 'created': '2025-07-23T17:12:44', 'last-modified': '2025-07-23T17:12:44', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.421392'} 2025-07-23 17:12:44,892 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-23 17:12:44,898 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-55648472', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '8268001356544099460', 'uuid-lslong': '12257286272019716229'}, 'enable': 'true', 'created': '2025-07-23T17:12:44', 'last-modified': '2025-07-23T17:12:44', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.441199'} 2025-07-23 17:12:44,905 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-23 17:12:44,913 - DEBUG - Control-node 10.0.0.73 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-55648472', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '8268001356544099460', 'uuid-lslong': '12257286272019716229'}, 'enable': 'true', 'created': '2025-07-23T17:12:44', 'last-modified': '2025-07-23T17:12:44', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.451747'} 2025-07-23 17:12:44,922 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-23 17:12:44,923 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn-31775303 passed 2025-07-23 17:12:44,923 - DEBUG - ====Verifying policy data for ctest-vn-31775303 in API_Server ====== 2025-07-23 17:12:44,923 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-07-23 17:12:44,930 - DEBUG - Requesting: http://10.0.0.23:8082/domain/85afe8be-97e7-4e27-9667-ee7272dde692 2025-07-23 17:12:44,944 - DEBUG - Requesting: http://10.0.0.23:8082/project/1d70cd1f-ffcb-4d19-b486-a265a2179d49 2025-07-23 17:12:45,039 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/72bdd76d-9342-4484-aa1a-a0ec2a01d885 2025-07-23 17:12:45,050 - DEBUG - =>VN ctest-vn-31775303 has no policy to be verified 2025-07-23 17:12:45,050 - DEBUG - Verifying the vn in opserver 2025-07-23 17:12:45,050 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303 virtual network link through opserver 10.0.0.23 2025-07-23 17:12:45,050 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-07-23 17:12:45,108 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303?flat'} 2025-07-23 17:12:45,108 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303 virtual network link through opserver 10.0.0.249 2025-07-23 17:12:45,108 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2025-07-23 17:12:45,171 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303?flat'} 2025-07-23 17:12:45,171 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303 virtual network link through opserver 10.0.0.73 2025-07-23 17:12:45,171 - DEBUG - Requesting: http://10.0.0.73:8081/analytics/uves/virtual-networks 2025-07-23 17:12:45,227 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303', 'href': 'http://10.0.0.73:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303?flat'} 2025-07-23 17:12:45,227 - INFO - Validated that VN default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303 is found in opserver 2025-07-23 17:12:45,269 - DEBUG - Do not have enough data to verify VN in agent 2025-07-23 17:12:45,288 - DEBUG - VRF ids for VN ctest-vn-31775303: {} 2025-07-23 17:12:45,288 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3704-2', 'an-jenkins-deploy-platform-ansible-os-3704-1']} 2025-07-23 17:12:45,288 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3704-2', 'an-jenkins-deploy-platform-ansible-os-3704-1']} 2025-07-23 17:12:46,505 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3704-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3704-2) 2025-07-23 17:12:46,505 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3704-2', 'an-jenkins-deploy-platform-ansible-os-3704-1']} 2025-07-23 17:12:47,976 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3704-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3704-2) 2025-07-23 17:12:47,976 - INFO - Waiting for VM ctest-TestMtuProbe-55648472-23482951 to be up.. 2025-07-23 17:12:48,070 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-07-23 17:12:53,155 - DEBUG - VM is in ACTIVE state now 2025-07-23 17:12:53,155 - INFO - VM name : ctest-TestMtuProbe-55648472-23482951 2025-07-23 17:12:53,247 - DEBUG - VM ctest-TestMtuProbe-55648472-23482951 ID is 1d72fb19-06e3-44ca-9658-7c05b7c4857e 2025-07-23 17:12:53,248 - DEBUG - VM ctest-TestMtuProbe-55648472-23482951 launched on Node an-jenkins-deploy-platform-ansible-os-3704-2 2025-07-23 17:12:53,343 - DEBUG - Requesting: http://10.0.0.73:8082/virtual-machine/1d72fb19-06e3-44ca-9658-7c05b7c4857e 2025-07-23 17:12:53,377 - DEBUG - Requesting: http://10.0.0.73:8082/virtual-machine-interface/bbb83a18-4ac6-425e-aa2e-eff67296253b 2025-07-23 17:12:56,646 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1038ms') 2025-07-23 17:12:56,646 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-23482951 failed! 2025-07-23 17:12:56,665 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303 is 59.93.191.193 and allocation pool is NOT set 2025-07-23 17:13:00,744 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-07-23 17:13:00,745 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-23482951 failed! 2025-07-23 17:13:00,762 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303 is 59.93.191.193 and allocation pool is NOT set 2025-07-23 17:13:04,834 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1004ms') 2025-07-23 17:13:04,834 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-23482951 failed! 2025-07-23 17:13:04,850 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303 is 59.93.191.193 and allocation pool is NOT set 2025-07-23 17:13:08,932 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-07-23 17:13:08,932 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-23482951 failed! 2025-07-23 17:13:08,950 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303 is 59.93.191.193 and allocation pool is NOT set 2025-07-23 17:13:13,031 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms') 2025-07-23 17:13:13,031 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-23482951 failed! 2025-07-23 17:13:13,046 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303 is 59.93.191.193 and allocation pool is NOT set 2025-07-23 17:13:17,128 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-07-23 17:13:17,128 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-23482951 failed! 2025-07-23 17:13:17,145 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303 is 59.93.191.193 and allocation pool is NOT set 2025-07-23 17:13:21,228 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-07-23 17:13:21,228 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-23482951 failed! 2025-07-23 17:13:21,247 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303 is 59.93.191.193 and allocation pool is NOT set 2025-07-23 17:13:25,316 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1008ms') 2025-07-23 17:13:25,316 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-23482951 failed! 2025-07-23 17:13:25,331 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303 is 59.93.191.193 and allocation pool is NOT set 2025-07-23 17:13:29,419 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-07-23 17:13:29,419 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-23482951 failed! 2025-07-23 17:13:29,438 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303 is 59.93.191.193 and allocation pool is NOT set 2025-07-23 17:13:33,510 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1004ms') 2025-07-23 17:13:33,511 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-23482951 failed! 2025-07-23 17:13:33,525 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-55648472:ctest-vn-31775303 is 59.93.191.193 and allocation pool is NOT set 2025-07-23 17:13:35,593 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=7.91 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.918 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.918/4.413/7.909/3.495 ms') 2025-07-23 17:13:35,593 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-55648472-23482951 passed 2025-07-23 17:13:35,705 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:13:35,705 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-55648472-23482951, IP 59.93.191.195, Port 22 2025-07-23 17:13:35,772 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-23 17:13:35,859 - DEBUG - VM ctest-TestMtuProbe-55648472-23482951 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-23 17:13:40,859 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:13:40,860 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-55648472-23482951, IP 59.93.191.195, Port 22 2025-07-23 17:13:40,926 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-23 17:13:41,005 - DEBUG - VM ctest-TestMtuProbe-55648472-23482951 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-23 17:13:46,006 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:13:46,006 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-55648472-23482951, IP 59.93.191.195, Port 22 2025-07-23 17:13:46,075 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-23 17:13:46,157 - DEBUG - VM ctest-TestMtuProbe-55648472-23482951 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-23 17:13:51,157 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:13:51,158 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-55648472-23482951, IP 59.93.191.195, Port 22 2025-07-23 17:13:51,233 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-07-23 17:13:51,313 - DEBUG - VM ctest-TestMtuProbe-55648472-23482951 is NOT ready for SSH connections, VM status: ACTIVE 2025-07-23 17:13:56,314 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:13:56,315 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-55648472-23482951, IP 59.93.191.195, Port 22 2025-07-23 17:13:56,487 - DEBUG - VM ctest-TestMtuProbe-55648472-23482951 is ready for SSH connections 2025-07-23 17:13:56,487 - INFO - Waiting for VM ctest-TestMtuProbe-55648472-33099025 to be up.. 2025-07-23 17:13:56,577 - DEBUG - VM is in ACTIVE state now 2025-07-23 17:13:56,577 - INFO - VM name : ctest-TestMtuProbe-55648472-33099025 2025-07-23 17:13:56,659 - DEBUG - VM ctest-TestMtuProbe-55648472-33099025 ID is 8821ebdf-a9bb-4e33-89d3-ee51953b8a11 2025-07-23 17:13:56,660 - DEBUG - VM ctest-TestMtuProbe-55648472-33099025 launched on Node an-jenkins-deploy-platform-ansible-os-3704-2 2025-07-23 17:13:56,740 - DEBUG - Requesting: http://10.0.0.73:8082/virtual-machine/8821ebdf-a9bb-4e33-89d3-ee51953b8a11 2025-07-23 17:13:56,751 - DEBUG - Requesting: http://10.0.0.73:8082/virtual-machine-interface/39c05d1a-7a01-45c0-ab20-b4646227fe21 2025-07-23 17:13:57,955 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=5.95 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.505 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.505/3.227/5.949/2.722 ms') 2025-07-23 17:13:57,955 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-TestMtuProbe-55648472-33099025 passed 2025-07-23 17:13:58,072 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:13:58,073 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-55648472-33099025, IP 59.93.191.196, Port 22 2025-07-23 17:13:58,238 - DEBUG - VM ctest-TestMtuProbe-55648472-33099025 is ready for SSH connections 2025-07-23 17:13:58,239 - DEBUG - Running remote_cmd, Cmd : echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:13:58,239 - DEBUG - echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-23 17:14:19,561 - DEBUG - None 2025-07-23 17:14:19,561 - DEBUG - Running remote_cmd, Cmd : echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:14:19,561 - DEBUG - echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-23 17:14:57,320 - DEBUG - None 2025-07-23 17:14:57,320 - INFO - {'echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-23 17:14:57,320 - INFO - {'echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-23 17:14:57,320 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:14:57,320 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 59.93.191.196, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:14:57,320 - DEBUG - ping -s 56 -c 3 -W 1 59.93.191.196 2025-07-23 17:14:59,798 - DEBUG - PING 59.93.191.196 (59.93.191.196) 56(84) bytes of data. 64 bytes from 59.93.191.196: icmp_req=1 ttl=64 time=7.32 ms 64 bytes from 59.93.191.196: icmp_req=2 ttl=64 time=0.808 ms 64 bytes from 59.93.191.196: icmp_req=3 ttl=64 time=0.724 ms --- 59.93.191.196 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 0.724/2.952/7.326/3.093 ms 2025-07-23 17:14:59,798 - INFO - Ping to IP 59.93.191.196 from VM ctest-TestMtuProbe-55648472-23482951 passed 2025-07-23 17:14:59,799 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:14:59,799 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:14:59,799 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile & 2025-07-23 17:15:00,402 - DEBUG - total 132 drwxr-xr-x 11 ubuntu ubuntu 4096 Dec 29 2017 . drwxr-xr-x 3 root root 4096 Jan 4 2013 .. -rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history -rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout -rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache -rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins -rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla -rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh -rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh -rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0 -rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz -rw------- 1 root root 4695 Dec 17 2017 .viminfo -rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority -rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors 2025-07-23 17:15:00,402 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:00,402 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile & 2025-07-23 17:15:00,879 - DEBUG - None 2025-07-23 17:15:00,879 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 59.93.191.196 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:00,879 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 59.93.191.196 5201 < testfile 2025-07-23 17:15:03,438 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.000770364 s, 130 kB/s 2025-07-23 17:15:03,438 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:03,438 - DEBUG - ls -l testfile 2025-07-23 17:15:03,767 - DEBUG - -rw-r--r-- 1 root root 100 Jul 23 17:15 testfile 2025-07-23 17:15:03,767 - INFO - File transfer verification for file size 100 passed on the VM ctest-TestMtuProbe-55648472-33099025 2025-07-23 17:15:03,767 - DEBUG - Running remote_cmd, Cmd : echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:03,767 - DEBUG - echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-23 17:15:04,205 - DEBUG - None 2025-07-23 17:15:04,205 - DEBUG - Running remote_cmd, Cmd : echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:04,205 - DEBUG - echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-23 17:15:04,630 - DEBUG - None 2025-07-23 17:15:04,630 - INFO - {'echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-23 17:15:04,630 - INFO - {'echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-23 17:15:04,630 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:15:04,630 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 59.93.191.196, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:04,630 - DEBUG - ping -s 56 -c 3 -W 1 59.93.191.196 2025-07-23 17:15:07,099 - DEBUG - PING 59.93.191.196 (59.93.191.196) 56(84) bytes of data. 64 bytes from 59.93.191.196: icmp_req=1 ttl=64 time=2.06 ms 64 bytes from 59.93.191.196: icmp_req=2 ttl=64 time=0.883 ms 64 bytes from 59.93.191.196: icmp_req=3 ttl=64 time=1.79 ms --- 59.93.191.196 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 0.883/1.583/2.068/0.507 ms 2025-07-23 17:15:07,099 - INFO - Ping to IP 59.93.191.196 from VM ctest-TestMtuProbe-55648472-23482951 passed 2025-07-23 17:15:07,099 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:15:07,099 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:07,099 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile & 2025-07-23 17:15:07,696 - DEBUG - total 132 drwxr-xr-x 11 ubuntu ubuntu 4096 Jul 23 17:15 . drwxr-xr-x 3 root root 4096 Jan 4 2013 .. -rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history -rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout -rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache -rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins -rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla -rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh -rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh -rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0 -rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz -rw------- 1 root root 4695 Dec 17 2017 .viminfo -rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority -rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors 2025-07-23 17:15:07,696 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:07,696 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile & 2025-07-23 17:15:08,161 - DEBUG - None 2025-07-23 17:15:08,161 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10002 59.93.191.196 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:08,161 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10002 59.93.191.196 5201 < testfile 2025-07-23 17:15:10,694 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.000959317 s, 104 kB/s 2025-07-23 17:15:10,694 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:10,694 - DEBUG - ls -l testfile 2025-07-23 17:15:11,034 - DEBUG - -rw-r--r-- 1 root root 100 Jul 23 17:15 testfile 2025-07-23 17:15:11,034 - INFO - File transfer verification for file size 100 passed on the VM ctest-TestMtuProbe-55648472-33099025 2025-07-23 17:15:11,035 - DEBUG - Running remote_cmd, Cmd : echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:11,035 - DEBUG - echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-23 17:15:11,466 - DEBUG - None 2025-07-23 17:15:11,466 - DEBUG - Running remote_cmd, Cmd : echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:11,466 - DEBUG - echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing 2025-07-23 17:15:11,879 - DEBUG - None 2025-07-23 17:15:11,879 - INFO - {'echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-23 17:15:11,879 - INFO - {'echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing': None} 2025-07-23 17:15:11,880 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:15:11,880 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 59.93.191.196, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:11,880 - DEBUG - ping -s 56 -c 3 -W 1 59.93.191.196 2025-07-23 17:15:14,341 - DEBUG - PING 59.93.191.196 (59.93.191.196) 56(84) bytes of data. 64 bytes from 59.93.191.196: icmp_req=1 ttl=64 time=1.84 ms 64 bytes from 59.93.191.196: icmp_req=2 ttl=64 time=0.876 ms 64 bytes from 59.93.191.196: icmp_req=3 ttl=64 time=0.759 ms --- 59.93.191.196 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 0.759/1.159/1.844/0.487 ms 2025-07-23 17:15:14,341 - INFO - Ping to IP 59.93.191.196 from VM ctest-TestMtuProbe-55648472-23482951 passed 2025-07-23 17:15:14,341 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-07-23 17:15:14,342 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:14,342 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile & 2025-07-23 17:15:14,925 - DEBUG - total 132 drwxr-xr-x 11 ubuntu ubuntu 4096 Jul 23 17:15 . drwxr-xr-x 3 root root 4096 Jan 4 2013 .. -rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history -rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout -rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache -rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins -rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla -rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh -rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh -rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0 -rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz -rw------- 1 root root 4695 Dec 17 2017 .viminfo -rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority -rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors 2025-07-23 17:15:14,925 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:14,925 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile & 2025-07-23 17:15:15,368 - DEBUG - None 2025-07-23 17:15:15,368 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 59.93.191.196 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:15,368 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 59.93.191.196 5201 < testfile 2025-07-23 17:15:17,973 - DEBUG - 1+0 records in 1+0 records out 100 bytes (100 B) copied, 0.00394573 s, 25.3 kB/s 2025-07-23 17:15:17,973 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.130, gateway password: c0ntrail123 2025-07-23 17:15:17,973 - DEBUG - ls -l testfile 2025-07-23 17:15:18,306 - DEBUG - -rw-r--r-- 1 root root 100 Jul 23 17:15 testfile 2025-07-23 17:15:18,306 - INFO - File transfer verification for file size 100 passed on the VM ctest-TestMtuProbe-55648472-33099025 2025-07-23 17:15:18,307 - INFO - Deleting VM ctest-TestMtuProbe-55648472-33099025 2025-07-23 17:15:18,371 - INFO - Deleting VM ctest-TestMtuProbe-55648472-23482951 2025-07-23 17:15:18,434 - INFO - Deleting VN ctest-vn-31775303 2025-07-23 17:15:18,481 - DEBUG - VN 72bdd76d-9342-4484-aa1a-a0ec2a01d885 still in use: Unable to complete operation on network 72bdd76d-9342-4484-aa1a-a0ec2a01d885. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-c2d7660e-dc4c-4898-a377-6e71e6e01d50'] 2025-07-23 17:15:18,481 - WARNING - Deleting VN ctest-vn-31775303 failed..Will retry 2025-07-23 17:15:20,682 - DEBUG - Response for deleting network () 2025-07-23 17:15:20,683 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/5d6ca6be-3574-4797-9987-d53b43df0d23 2025-07-23 17:15:20,689 - DEBUG - Response Code: 404 2025-07-23 17:15:20,689 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-07-23 17:15:20,705 - DEBUG - Requesting: http://10.0.0.23:8082/domain/85afe8be-97e7-4e27-9667-ee7272dde692 2025-07-23 17:15:20,765 - DEBUG - Requesting: http://10.0.0.23:8082/project/1d70cd1f-ffcb-4d19-b486-a265a2179d49 2025-07-23 17:15:20,891 - INFO - Validated that VN ctest-vn-31775303 is not found in API Server 2025-07-23 17:15:20,918 - DEBUG - VN ctest-vn-31775303 is not present in Agent 10.0.0.50 2025-07-23 17:15:20,937 - DEBUG - VN ctest-vn-31775303 is not present in Agent 10.0.0.130 2025-07-23 17:15:20,937 - INFO - Validated that VN ctest-vn-31775303 is not in any agent 2025-07-23 17:15:20,993 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn-31775303 info 2025-07-23 17:15:22,315 - DEBUG - No XMPP flaps were noticed during the test 2025-07-23 17:15:22,315 - INFO - END TEST : test_mtu_probe_intra_node : PASSED[0:02:40] 2025-07-23 17:15:22,315 - INFO - -------------------------------------------------------------------------------- 2025-07-23 17:15:23,089 - INFO - Deleted project: ctest-TestMtuProbe-55648472, ID : 1d70cd1f-ffcb-4d19-b486-a265a2179d49