2025-07-31 04:54:47,769 - INFO - Domain Default found not creating
2025-07-31 04:54:47,950 - INFO - Project ctest-TestMtuProbe-15541328 not found, creating it
2025-07-31 04:54:48,668 - INFO - Created Project:ctest-TestMtuProbe-15541328, ID : e8822bde-b9be-4614-8a35-0e1e4b75b1f1
2025-07-31 04:54:50,470 - INFO - ================================================================================
2025-07-31 04:54:50,471 - INFO - STARTING TEST : test_mtu_probe_inter_node
2025-07-31 04:54:50,471 - INFO - TEST DESCRIPTION :
Description:
check traffic configuring different tcp_mtu_probe vlaues on vm
MTU probing values: 0, 1, 2
0 - Disabled (default)
1 - Disabled by default, Enabled only when ICMP black hole detected
2 - Always enabled, using the initial MSS value of tcp_base_mss
system internally enables bit values for point 1&2
For test purpose,
Idea is to check traffic works fine on configuring values
Vm's across compute node
steps:
1. create 1 VN and launch 2 VMs across compute node
2. client and server VMs across compute node
3. configure tcp_mtu_probe to 0, 1, 2
4. send icmp traffic assert if ping fails
5. send tcp traffic assert if tcp traffic fails
Pass criteria:
1. If icmp/tcp traffic work fine configuring tcp_mtu_probe values
2025-07-31 04:54:51,891 - DEBUG - Nothing to compare xmpp stats {'10.0.0.27': {'10.20.0.193': '0', '10.20.0.18': '0'}, '10.0.0.57': {'10.20.0.18': '0', '10.20.0.193': '0'}} with
2025-07-31 04:54:51,891 - INFO - Initial checks done. Running the testcase now
2025-07-31 04:54:51,891 - INFO -
2025-07-31 04:54:51,922 - DEBUG - Services list from nova: [, , , ]
2025-07-31 04:54:51,923 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']}
2025-07-31 04:54:52,855 - DEBUG - Response for create_network : {'network': {'id': '57f03fbd-2014-4b85-acb1-0d3ccb0e6724', 'name': 'ctest-vn-93417578', 'tenant_id': 'e8822bdeb9be46148a350e1e4b75b1f1', 'project_id': 'e8822bdeb9be46148a350e1e4b75b1f1', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestMtuProbe-15541328', 'ctest-vn-93417578'], 'port_security_enabled': True, 'description': ''}}
2025-07-31 04:54:53,128 - DEBUG - Response for create_subnet : {'subnet': {'id': '3bec82a0-bff0-4700-8e34-a34725e4c65e', 'name': '', 'tenant_id': 'e8822bdeb9be46148a350e1e4b75b1f1', 'network_id': '57f03fbd-2014-4b85-acb1-0d3ccb0e6724', 'ip_version': 4, 'cidr': '161.94.71.0/26', 'allocation_pools': [{'start': '161.94.71.2', 'end': '161.94.71.62'}], 'gateway_ip': '161.94.71.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '161.94.71.2', 'tags': [], 'project_id': 'e8822bdeb9be46148a350e1e4b75b1f1'}}
2025-07-31 04:54:53,153 - INFO - Created VN ctest-vn-93417578
2025-07-31 04:54:53,167 - DEBUG - VN ctest-vn-93417578 UUID is 57f03fbd-2014-4b85-acb1-0d3ccb0e6724
2025-07-31 04:54:53,252 - DEBUG - Requesting: http://10.0.0.41:8082/domains
2025-07-31 04:54:53,587 - DEBUG - Requesting: http://10.0.0.41:8082/domains
2025-07-31 04:54:53,640 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7
2025-07-31 04:54:53,683 - DEBUG - Requesting: http://10.0.0.41:8082/project/e8822bde-b9be-4614-8a35-0e1e4b75b1f1
2025-07-31 04:54:53,815 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/57f03fbd-2014-4b85-acb1-0d3ccb0e6724
2025-07-31 04:54:53,841 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/57f03fbd-2014-4b85-acb1-0d3ccb0e6724
2025-07-31 04:54:53,859 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/3a818f26-b148-455f-8164-2a9ad49a209f
2025-07-31 04:54:53,876 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/3a818f26-b148-455f-8164-2a9ad49a209f
2025-07-31 04:54:53,891 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/3ef1e4e9-8111-4c1b-aaf2-494466572728
2025-07-31 04:54:53,902 - DEBUG - Route Targets: ['target:64512:8000005']
2025-07-31 04:54:53,903 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/57f03fbd-2014-4b85-acb1-0d3ccb0e6724
2025-07-31 04:54:53,921 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/3a818f26-b148-455f-8164-2a9ad49a209f
2025-07-31 04:54:53,944 - INFO - Verified VN network id 11 for VN 57f03fbd-2014-4b85-acb1-0d3ccb0e6724
2025-07-31 04:54:53,944 - INFO - Verifications in API Server for VN ctest-vn-93417578 passed
2025-07-31 04:54:53,944 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/57f03fbd-2014-4b85-acb1-0d3ccb0e6724
2025-07-31 04:54:53,961 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/3a818f26-b148-455f-8164-2a9ad49a209f
2025-07-31 04:54:53,977 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/3a818f26-b148-455f-8164-2a9ad49a209f
2025-07-31 04:54:53,990 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/3ef1e4e9-8111-4c1b-aaf2-494466572728
2025-07-31 04:54:54,013 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-15541328', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6336634757229857669', 'uuid-lslong': '12443741800157308708'}, 'enable': 'true', 'created': '2025-07-31T04:54:52', 'last-modified': '2025-07-31T04:54:53', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.950413'}
2025-07-31 04:54:54,021 - DEBUG - Route Targets: ['target:64512:8000005']
2025-07-31 04:54:54,030 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-15541328', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6336634757229857669', 'uuid-lslong': '12443741800157308708'}, 'enable': 'true', 'created': '2025-07-31T04:54:52', 'last-modified': '2025-07-31T04:54:53', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.966223'}
2025-07-31 04:54:54,038 - DEBUG - Route Targets: ['target:64512:8000005']
2025-07-31 04:54:54,048 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-15541328', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6336634757229857669', 'uuid-lslong': '12443741800157308708'}, 'enable': 'true', 'created': '2025-07-31T04:54:52', 'last-modified': '2025-07-31T04:54:53', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.984584'}
2025-07-31 04:54:54,059 - DEBUG - Route Targets: ['target:64512:8000005']
2025-07-31 04:54:54,059 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn-93417578 passed
2025-07-31 04:54:54,060 - DEBUG - ====Verifying policy data for ctest-vn-93417578 in API_Server ======
2025-07-31 04:54:54,060 - DEBUG - Requesting: http://10.0.0.41:8082/domains
2025-07-31 04:54:54,075 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7
2025-07-31 04:54:54,103 - DEBUG - Requesting: http://10.0.0.41:8082/project/e8822bde-b9be-4614-8a35-0e1e4b75b1f1
2025-07-31 04:54:54,250 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/57f03fbd-2014-4b85-acb1-0d3ccb0e6724
2025-07-31 04:54:54,267 - DEBUG - =>VN ctest-vn-93417578 has no policy to be verified
2025-07-31 04:54:54,267 - DEBUG - Verifying the vn in opserver
2025-07-31 04:54:54,268 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 virtual network link through opserver 10.0.0.41
2025-07-31 04:54:54,268 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks
2025-07-31 04:54:54,593 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks
2025-07-31 04:54:54,664 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578?flat'}
2025-07-31 04:54:54,664 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 virtual network link through opserver 10.0.0.38
2025-07-31 04:54:54,665 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks
2025-07-31 04:54:54,963 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks
2025-07-31 04:54:55,044 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578?flat'}
2025-07-31 04:54:55,044 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 virtual network link through opserver 10.0.0.61
2025-07-31 04:54:55,044 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks
2025-07-31 04:54:55,365 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks
2025-07-31 04:54:55,423 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578?flat'}
2025-07-31 04:54:55,423 - INFO - Validated that VN default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is found in opserver
2025-07-31 04:54:55,458 - DEBUG - Do not have enough data to verify VN in agent
2025-07-31 04:54:55,477 - DEBUG - VRF ids for VN ctest-vn-93417578: {}
2025-07-31 04:54:55,478 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']}
2025-07-31 04:54:55,478 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']}
2025-07-31 04:54:56,757 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-1)
2025-07-31 04:54:56,757 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']}
2025-07-31 04:54:58,925 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-2)
2025-07-31 04:54:58,925 - INFO - Waiting for VM ctest-TestMtuProbe-15541328-27761176 to be up..
2025-07-31 04:54:59,022 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2025-07-31 04:55:04,179 - DEBUG - VM is in ACTIVE state now
2025-07-31 04:55:04,179 - INFO - VM name : ctest-TestMtuProbe-15541328-27761176
2025-07-31 04:55:04,312 - DEBUG - VM ctest-TestMtuProbe-15541328-27761176 ID is 044c8598-01ad-4883-afc8-6a837d0239e4
2025-07-31 04:55:04,355 - DEBUG - VM ctest-TestMtuProbe-15541328-27761176 launched on Node an-jenkins-deploy-platform-ansible-os-3732-1
2025-07-31 04:55:04,504 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/044c8598-01ad-4883-afc8-6a837d0239e4
2025-07-31 04:55:04,817 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/044c8598-01ad-4883-afc8-6a837d0239e4
2025-07-31 04:55:04,871 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/36fedd06-a977-44ff-83d1-bdbcf3632683
2025-07-31 04:55:08,171 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms')
2025-07-31 04:55:08,172 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed!
2025-07-31 04:55:08,196 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set
2025-07-31 04:55:12,301 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1034ms')
2025-07-31 04:55:12,301 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed!
2025-07-31 04:55:12,329 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set
2025-07-31 04:55:16,425 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1024ms')
2025-07-31 04:55:16,426 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed!
2025-07-31 04:55:16,455 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set
2025-07-31 04:55:20,556 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1024ms')
2025-07-31 04:55:20,557 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed!
2025-07-31 04:55:20,580 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set
2025-07-31 04:55:24,654 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1007ms')
2025-07-31 04:55:24,654 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed!
2025-07-31 04:55:24,681 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set
2025-07-31 04:55:28,784 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1035ms')
2025-07-31 04:55:28,784 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed!
2025-07-31 04:55:28,805 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set
2025-07-31 04:55:32,876 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms')
2025-07-31 04:55:32,876 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed!
2025-07-31 04:55:32,895 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set
2025-07-31 04:55:36,973 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms')
2025-07-31 04:55:36,974 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed!
2025-07-31 04:55:37,000 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set
2025-07-31 04:55:41,102 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1032ms')
2025-07-31 04:55:41,102 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 failed!
2025-07-31 04:55:41,128 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-93417578 is 161.94.71.1 and allocation pool is NOT set
2025-07-31 04:55:43,196 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=7.49 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=4.34 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 4.343/5.917/7.491/1.574 ms')
2025-07-31 04:55:43,197 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-27761176 passed
2025-07-31 04:55:43,327 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 04:55:43,327 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-27761176, IP 161.94.71.3, Port 22
2025-07-31 04:55:43,390 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-07-31 04:55:43,567 - DEBUG - VM ctest-TestMtuProbe-15541328-27761176 is NOT ready for SSH connections, VM status: ACTIVE
2025-07-31 04:55:48,569 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 04:55:48,569 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-27761176, IP 161.94.71.3, Port 22
2025-07-31 04:55:48,632 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-07-31 04:55:48,815 - DEBUG - VM ctest-TestMtuProbe-15541328-27761176 is NOT ready for SSH connections, VM status: ACTIVE
2025-07-31 04:55:53,816 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 04:55:53,816 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-27761176, IP 161.94.71.3, Port 22
2025-07-31 04:55:53,889 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-07-31 04:55:54,066 - DEBUG - VM ctest-TestMtuProbe-15541328-27761176 is NOT ready for SSH connections, VM status: ACTIVE
2025-07-31 04:55:59,067 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 04:55:59,067 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-27761176, IP 161.94.71.3, Port 22
2025-07-31 04:55:59,133 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-07-31 04:55:59,304 - DEBUG - VM ctest-TestMtuProbe-15541328-27761176 is NOT ready for SSH connections, VM status: ACTIVE
2025-07-31 04:56:04,304 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 04:56:04,305 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-27761176, IP 161.94.71.3, Port 22
2025-07-31 04:56:04,463 - DEBUG - VM ctest-TestMtuProbe-15541328-27761176 is ready for SSH connections
2025-07-31 04:56:04,463 - INFO - Waiting for VM ctest-TestMtuProbe-15541328-14386499 to be up..
2025-07-31 04:56:04,558 - DEBUG - VM is in ACTIVE state now
2025-07-31 04:56:04,558 - INFO - VM name : ctest-TestMtuProbe-15541328-14386499
2025-07-31 04:56:04,640 - DEBUG - VM ctest-TestMtuProbe-15541328-14386499 ID is 3dfb85f0-8de8-4200-a578-553d6f3575a0
2025-07-31 04:56:04,641 - DEBUG - VM ctest-TestMtuProbe-15541328-14386499 launched on Node an-jenkins-deploy-platform-ansible-os-3732-2
2025-07-31 04:56:04,733 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/3dfb85f0-8de8-4200-a578-553d6f3575a0
2025-07-31 04:56:04,747 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/77fefad4-1a29-46fb-a7ec-019f08f22475
2025-07-31 04:56:05,964 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=3.89 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.849 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.849/2.369/3.889/1.520 ms')
2025-07-31 04:56:05,964 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-14386499 passed
2025-07-31 04:56:06,084 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 04:56:06,084 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-14386499, IP 161.94.71.4, Port 22
2025-07-31 04:56:06,156 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-07-31 04:56:06,259 - DEBUG - VM ctest-TestMtuProbe-15541328-14386499 is NOT ready for SSH connections, VM status: ACTIVE
2025-07-31 04:56:11,260 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 04:56:11,261 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-14386499, IP 161.94.71.4, Port 22
2025-07-31 04:56:11,442 - DEBUG - VM ctest-TestMtuProbe-15541328-14386499 is ready for SSH connections
2025-07-31 04:56:11,442 - DEBUG - Running remote_cmd, Cmd : echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 04:56:11,443 - DEBUG - echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing
2025-07-31 04:57:00,709 - DEBUG - None
2025-07-31 04:57:00,709 - DEBUG - Running remote_cmd, Cmd : echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:57:00,709 - DEBUG - echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing
2025-07-31 04:57:46,563 - DEBUG - None
2025-07-31 04:57:46,563 - INFO - {'echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing': None}
2025-07-31 04:57:46,563 - INFO - {'echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing': None}
2025-07-31 04:57:46,564 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 04:57:46,564 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 161.94.71.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 04:57:46,565 - DEBUG - ping -s 56 -c 3 -W 1 161.94.71.4
2025-07-31 04:57:49,082 - DEBUG - PING 161.94.71.4 (161.94.71.4) 56(84) bytes of data.
64 bytes from 161.94.71.4: icmp_req=1 ttl=64 time=11.7 ms
64 bytes from 161.94.71.4: icmp_req=2 ttl=64 time=1.18 ms
64 bytes from 161.94.71.4: icmp_req=3 ttl=64 time=1.02 ms
--- 161.94.71.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.023/4.647/11.737/5.013 ms
2025-07-31 04:57:49,082 - INFO - Ping to IP 161.94.71.4 from VM ctest-TestMtuProbe-15541328-27761176 passed
2025-07-31 04:57:49,083 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 04:57:49,083 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:57:49,084 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile &
2025-07-31 04:57:49,776 - DEBUG - total 132
drwxr-xr-x 11 ubuntu ubuntu 4096 Dec 29 2017 .
drwxr-xr-x 3 root root 4096 Jan 4 2013 ..
-rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history
-rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout
-rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc
drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache
-rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh
drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig
drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm
drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins
-rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst
drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla
-rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh
-rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr
drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh
-rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic
drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0
-rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz
-rw------- 1 root root 4695 Dec 17 2017 .viminfo
-rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority
-rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors
2025-07-31 04:57:49,776 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:57:49,776 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile &
2025-07-31 04:57:50,312 - DEBUG - None
2025-07-31 04:57:50,313 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 04:57:50,313 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 161.94.71.4 5201 < testfile
2025-07-31 04:57:55,870 - DEBUG - 1+0 records in
1+0 records out
100 bytes (100 B) copied, 0.000870844 s, 115 kB/s
2025-07-31 04:57:55,870 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:57:55,870 - DEBUG - ls -l testfile
2025-07-31 04:57:56,282 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:57:56,282 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:57:59,283 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:57:59,284 - DEBUG - ls -l testfile
2025-07-31 04:57:59,664 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:57:59,664 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:02,665 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:02,666 - DEBUG - ls -l testfile
2025-07-31 04:58:03,075 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:03,075 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:06,077 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:06,077 - DEBUG - ls -l testfile
2025-07-31 04:58:06,508 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:06,509 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:09,510 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:09,511 - DEBUG - ls -l testfile
2025-07-31 04:58:09,889 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:09,889 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:12,890 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:12,891 - DEBUG - ls -l testfile
2025-07-31 04:58:13,285 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:13,286 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:16,287 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:16,287 - DEBUG - ls -l testfile
2025-07-31 04:58:16,709 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:16,710 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:19,710 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:19,711 - DEBUG - ls -l testfile
2025-07-31 04:58:20,109 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:20,109 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:23,110 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:23,111 - DEBUG - ls -l testfile
2025-07-31 04:58:23,466 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:23,467 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:26,467 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:26,467 - DEBUG - ls -l testfile
2025-07-31 04:58:26,877 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:26,878 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:29,879 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:29,880 - DEBUG - ls -l testfile
2025-07-31 04:58:30,227 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:30,227 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:30,228 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-15541328-14386499
2025-07-31 04:58:30,228 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 04:58:30,229 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 161.94.71.4 5201 < testfile
2025-07-31 04:58:32,836 - DEBUG - 1+0 records in
1+0 records out
100 bytes (100 B) copied, 0.00122353 s, 81.7 kB/s
2025-07-31 04:58:32,836 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:32,836 - DEBUG - ls -l testfile
2025-07-31 04:58:33,225 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:33,225 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:36,226 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:36,227 - DEBUG - ls -l testfile
2025-07-31 04:58:36,632 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:36,633 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:39,634 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:39,634 - DEBUG - ls -l testfile
2025-07-31 04:58:40,096 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:40,096 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:43,098 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:43,098 - DEBUG - ls -l testfile
2025-07-31 04:58:43,482 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:43,483 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:46,483 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:46,484 - DEBUG - ls -l testfile
2025-07-31 04:58:46,881 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:46,881 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:49,882 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:49,883 - DEBUG - ls -l testfile
2025-07-31 04:58:50,329 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:50,330 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:53,331 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:53,332 - DEBUG - ls -l testfile
2025-07-31 04:58:53,724 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:57 testfile
2025-07-31 04:58:53,724 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:58:56,725 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:56,725 - DEBUG - ls -l testfile
2025-07-31 04:58:57,088 - DEBUG - -rw-r--r-- 1 root root 100 Jul 31 04:58 testfile
2025-07-31 04:58:57,089 - INFO - File transfer verification for file size 100 passed on the VM ctest-TestMtuProbe-15541328-14386499
2025-07-31 04:58:57,089 - DEBUG - Running remote_cmd, Cmd : echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 04:58:57,090 - DEBUG - echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing
2025-07-31 04:58:57,615 - DEBUG - None
2025-07-31 04:58:57,616 - DEBUG - Running remote_cmd, Cmd : echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:58:57,616 - DEBUG - echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing
2025-07-31 04:58:58,704 - DEBUG - None
2025-07-31 04:58:58,705 - INFO - {'echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing': None}
2025-07-31 04:58:58,705 - INFO - {'echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing': None}
2025-07-31 04:58:58,705 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 04:58:58,705 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 161.94.71.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 04:58:58,705 - DEBUG - ping -s 56 -c 3 -W 1 161.94.71.4
2025-07-31 04:59:01,174 - DEBUG - PING 161.94.71.4 (161.94.71.4) 56(84) bytes of data.
64 bytes from 161.94.71.4: icmp_req=1 ttl=64 time=7.47 ms
64 bytes from 161.94.71.4: icmp_req=2 ttl=64 time=1.31 ms
64 bytes from 161.94.71.4: icmp_req=3 ttl=64 time=1.21 ms
--- 161.94.71.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.215/3.333/7.470/2.925 ms
2025-07-31 04:59:01,174 - INFO - Ping to IP 161.94.71.4 from VM ctest-TestMtuProbe-15541328-27761176 passed
2025-07-31 04:59:01,175 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 04:59:01,175 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:01,175 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile &
2025-07-31 04:59:01,986 - DEBUG - total 132
drwxr-xr-x 11 ubuntu ubuntu 4096 Jul 31 04:59 .
drwxr-xr-x 3 root root 4096 Jan 4 2013 ..
-rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history
-rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout
-rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc
drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache
-rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh
drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig
drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm
drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins
-rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst
drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla
-rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh
-rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr
drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh
-rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic
drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0
-rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz
-rw------- 1 root root 4695 Dec 17 2017 .viminfo
-rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority
-rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors
2025-07-31 04:59:01,986 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:01,986 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile &
2025-07-31 04:59:02,564 - DEBUG - None
2025-07-31 04:59:02,564 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10002 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 04:59:02,564 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10002 161.94.71.4 5201 < testfile
2025-07-31 04:59:08,103 - DEBUG - 1+0 records in
1+0 records out
100 bytes (100 B) copied, 0.000969596 s, 103 kB/s
2025-07-31 04:59:08,103 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:08,103 - DEBUG - ls -l testfile
2025-07-31 04:59:08,550 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:08,551 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:11,552 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:11,552 - DEBUG - ls -l testfile
2025-07-31 04:59:11,961 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:11,962 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:14,962 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:14,963 - DEBUG - ls -l testfile
2025-07-31 04:59:15,357 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:15,358 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:18,359 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:18,360 - DEBUG - ls -l testfile
2025-07-31 04:59:18,824 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:18,824 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:21,825 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:21,826 - DEBUG - ls -l testfile
2025-07-31 04:59:22,250 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:22,251 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:25,252 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:25,252 - DEBUG - ls -l testfile
2025-07-31 04:59:25,683 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:25,684 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:28,684 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:28,685 - DEBUG - ls -l testfile
2025-07-31 04:59:29,112 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:29,112 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:32,113 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:32,113 - DEBUG - ls -l testfile
2025-07-31 04:59:32,557 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:32,557 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:35,559 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:35,559 - DEBUG - ls -l testfile
2025-07-31 04:59:35,960 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:35,960 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:38,961 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:38,961 - DEBUG - ls -l testfile
2025-07-31 04:59:39,332 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:39,332 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:42,333 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:42,334 - DEBUG - ls -l testfile
2025-07-31 04:59:42,700 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:42,700 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:42,700 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-15541328-14386499
2025-07-31 04:59:42,701 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10002 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 04:59:42,701 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10002 161.94.71.4 5201 < testfile
2025-07-31 04:59:45,272 - DEBUG - 1+0 records in
1+0 records out
100 bytes (100 B) copied, 0.00103981 s, 96.2 kB/s
2025-07-31 04:59:45,273 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:45,273 - DEBUG - ls -l testfile
2025-07-31 04:59:45,693 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:45,694 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:48,694 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:48,695 - DEBUG - ls -l testfile
2025-07-31 04:59:49,089 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:49,090 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:52,090 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:52,091 - DEBUG - ls -l testfile
2025-07-31 04:59:52,528 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:52,528 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:55,536 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:55,537 - DEBUG - ls -l testfile
2025-07-31 04:59:55,907 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:55,908 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 04:59:58,908 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 04:59:58,909 - DEBUG - ls -l testfile
2025-07-31 04:59:59,291 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 04:59:59,291 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:02,292 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:02,293 - DEBUG - ls -l testfile
2025-07-31 05:00:02,715 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 05:00:02,715 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:05,716 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:05,716 - DEBUG - ls -l testfile
2025-07-31 05:00:06,083 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 05:00:06,083 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:09,084 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:09,085 - DEBUG - ls -l testfile
2025-07-31 05:00:09,434 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 04:59 testfile
2025-07-31 05:00:09,435 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:12,435 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:12,436 - DEBUG - ls -l testfile
2025-07-31 05:00:12,830 - DEBUG - -rw-r--r-- 1 root root 100 Jul 31 05:00 testfile
2025-07-31 05:00:12,830 - INFO - File transfer verification for file size 100 passed on the VM ctest-TestMtuProbe-15541328-14386499
2025-07-31 05:00:12,830 - DEBUG - Running remote_cmd, Cmd : echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:00:12,830 - DEBUG - echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing
2025-07-31 05:00:13,259 - DEBUG - None
2025-07-31 05:00:13,259 - DEBUG - Running remote_cmd, Cmd : echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:13,260 - DEBUG - echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing
2025-07-31 05:00:13,819 - DEBUG - None
2025-07-31 05:00:13,820 - INFO - {'echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing': None}
2025-07-31 05:00:13,820 - INFO - {'echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing': None}
2025-07-31 05:00:13,820 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:00:13,821 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 161.94.71.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:00:13,821 - DEBUG - ping -s 56 -c 3 -W 1 161.94.71.4
2025-07-31 05:00:16,307 - DEBUG - PING 161.94.71.4 (161.94.71.4) 56(84) bytes of data.
64 bytes from 161.94.71.4: icmp_req=1 ttl=64 time=2.97 ms
64 bytes from 161.94.71.4: icmp_req=2 ttl=64 time=1.71 ms
64 bytes from 161.94.71.4: icmp_req=3 ttl=64 time=1.36 ms
--- 161.94.71.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.363/2.019/2.978/0.694 ms
2025-07-31 05:00:16,308 - INFO - Ping to IP 161.94.71.4 from VM ctest-TestMtuProbe-15541328-27761176 passed
2025-07-31 05:00:16,308 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:00:16,308 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:16,308 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile &
2025-07-31 05:00:17,006 - DEBUG - total 132
drwxr-xr-x 11 ubuntu ubuntu 4096 Jul 31 05:00 .
drwxr-xr-x 3 root root 4096 Jan 4 2013 ..
-rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history
-rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout
-rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc
drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache
-rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh
drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig
drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm
drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins
-rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst
drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla
-rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh
-rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr
drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh
-rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic
drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0
-rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz
-rw------- 1 root root 4695 Dec 17 2017 .viminfo
-rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority
-rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors
2025-07-31 05:00:17,007 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:17,007 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile &
2025-07-31 05:00:17,552 - DEBUG - None
2025-07-31 05:00:17,553 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:00:17,553 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile
2025-07-31 05:00:23,165 - DEBUG - 1+0 records in
1+0 records out
100 bytes (100 B) copied, 0.00156089 s, 64.1 kB/s
2025-07-31 05:00:23,165 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:23,165 - DEBUG - ls -l testfile
2025-07-31 05:00:23,559 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:00:23,560 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:26,561 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:26,561 - DEBUG - ls -l testfile
2025-07-31 05:00:26,956 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:00:26,956 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:29,957 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:29,957 - DEBUG - ls -l testfile
2025-07-31 05:00:30,406 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:00:30,408 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:33,409 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:33,409 - DEBUG - ls -l testfile
2025-07-31 05:00:33,793 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:00:33,793 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:36,798 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:36,799 - DEBUG - ls -l testfile
2025-07-31 05:00:37,268 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:00:37,268 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:40,269 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:40,269 - DEBUG - ls -l testfile
2025-07-31 05:00:40,661 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:00:40,661 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:43,663 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:43,663 - DEBUG - ls -l testfile
2025-07-31 05:00:44,035 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:00:44,035 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:47,035 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:47,036 - DEBUG - ls -l testfile
2025-07-31 05:00:47,413 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:00:47,413 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:50,414 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:50,415 - DEBUG - ls -l testfile
2025-07-31 05:00:50,825 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:00:50,825 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:53,826 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:53,826 - DEBUG - ls -l testfile
2025-07-31 05:00:54,300 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:00:54,300 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:57,302 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:00:57,302 - DEBUG - ls -l testfile
2025-07-31 05:00:57,676 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:00:57,676 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:00:57,676 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-15541328-14386499
2025-07-31 05:00:57,677 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:00:57,677 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile
2025-07-31 05:01:00,249 - DEBUG - 1+0 records in
1+0 records out
100 bytes (100 B) copied, 0.00101222 s, 98.8 kB/s
2025-07-31 05:01:00,250 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:01:00,250 - DEBUG - ls -l testfile
2025-07-31 05:01:00,674 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:01:00,674 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:01:03,675 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:01:03,675 - DEBUG - ls -l testfile
2025-07-31 05:01:04,094 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:01:04,095 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:01:07,096 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:01:07,096 - DEBUG - ls -l testfile
2025-07-31 05:01:07,488 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:01:07,488 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:01:10,490 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:01:10,490 - DEBUG - ls -l testfile
2025-07-31 05:01:10,932 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:01:10,932 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:01:13,933 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:01:13,933 - DEBUG - ls -l testfile
2025-07-31 05:01:14,306 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:01:14,306 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:01:17,307 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:01:17,308 - DEBUG - ls -l testfile
2025-07-31 05:01:17,701 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:01:17,701 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:01:20,702 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:01:20,703 - DEBUG - ls -l testfile
2025-07-31 05:01:21,140 - DEBUG - -rw-r--r-- 1 root root 0 Jul 31 05:00 testfile
2025-07-31 05:01:21,140 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:01:24,140 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:01:24,141 - DEBUG - ls -l testfile
2025-07-31 05:01:24,507 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:01:24,507 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:01:27,508 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:01:27,508 - DEBUG - ls -l testfile
2025-07-31 05:01:27,883 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:01:27,883 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:01:30,884 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:01:30,885 - DEBUG - ls -l testfile
2025-07-31 05:01:31,263 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:01:31,263 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:01:34,264 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:01:34,265 - DEBUG - ls -l testfile
2025-07-31 05:01:34,636 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:01:34,636 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:01:34,637 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-15541328-14386499
2025-07-31 05:01:34,637 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:01:34,637 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile
2025-07-31 05:01:35,237 - DEBUG - 1+0 records in
1+0 records out
100 bytes (100 B) copied, 0.00103085 s, 97.0 kB/s
nc: bind failed: Address already in use
2025-07-31 05:01:35,238 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:01:35,238 - DEBUG - nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile
2025-07-31 05:01:35,847 - DEBUG - nc: bind failed: Address already in use
2025-07-31 05:01:41,848 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:01:41,848 - DEBUG - nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile
2025-07-31 05:01:42,331 - DEBUG - nc: bind failed: Address already in use
2025-07-31 05:01:48,333 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:01:48,334 - DEBUG - nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile
2025-07-31 05:01:49,253 - DEBUG - nc: bind failed: Address already in use
2025-07-31 05:01:55,254 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:01:55,255 - DEBUG - nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile
2025-07-31 05:02:00,825 - DEBUG - None
2025-07-31 05:02:00,825 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:00,825 - DEBUG - ls -l testfile
2025-07-31 05:02:01,294 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:01,294 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:04,296 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:04,296 - DEBUG - ls -l testfile
2025-07-31 05:02:04,751 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:04,751 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:07,752 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:07,753 - DEBUG - ls -l testfile
2025-07-31 05:02:08,165 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:08,165 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:11,165 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:11,166 - DEBUG - ls -l testfile
2025-07-31 05:02:11,530 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:11,531 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:14,532 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:14,533 - DEBUG - ls -l testfile
2025-07-31 05:02:14,925 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:14,925 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:17,926 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:17,926 - DEBUG - ls -l testfile
2025-07-31 05:02:18,318 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:18,319 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:21,320 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:21,320 - DEBUG - ls -l testfile
2025-07-31 05:02:21,809 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:21,809 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:24,810 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:24,811 - DEBUG - ls -l testfile
2025-07-31 05:02:25,202 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:25,203 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:28,204 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:28,204 - DEBUG - ls -l testfile
2025-07-31 05:02:28,556 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:28,556 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:31,556 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:31,557 - DEBUG - ls -l testfile
2025-07-31 05:02:31,997 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:31,998 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:34,999 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:34,999 - DEBUG - ls -l testfile
2025-07-31 05:02:35,361 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:35,361 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:35,361 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-15541328-14386499
2025-07-31 05:02:35,361 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:02:35,361 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile
2025-07-31 05:02:35,930 - DEBUG - 1+0 records in
1+0 records out
100 bytes (100 B) copied, 0.0113714 s, 8.8 kB/s
2025-07-31 05:02:35,930 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:35,931 - DEBUG - ls -l testfile
2025-07-31 05:02:36,391 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:36,391 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:39,392 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:39,393 - DEBUG - ls -l testfile
2025-07-31 05:02:39,779 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:39,779 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:42,779 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:42,780 - DEBUG - ls -l testfile
2025-07-31 05:02:43,151 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:43,151 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:46,153 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:46,153 - DEBUG - ls -l testfile
2025-07-31 05:02:46,525 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:46,527 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:49,528 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:49,529 - DEBUG - ls -l testfile
2025-07-31 05:02:49,865 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:49,865 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:52,867 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:52,867 - DEBUG - ls -l testfile
2025-07-31 05:02:53,265 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:53,265 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:56,266 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:56,267 - DEBUG - ls -l testfile
2025-07-31 05:02:56,656 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:02:56,657 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:02:59,658 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:02:59,659 - DEBUG - ls -l testfile
2025-07-31 05:03:00,033 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:00,033 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:03,034 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:03:03,034 - DEBUG - ls -l testfile
2025-07-31 05:03:03,426 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:03,426 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:06,428 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:03:06,429 - DEBUG - ls -l testfile
2025-07-31 05:03:06,838 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:06,839 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:09,840 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:03:09,841 - DEBUG - ls -l testfile
2025-07-31 05:03:10,225 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:10,225 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:10,225 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-15541328-14386499
2025-07-31 05:03:10,225 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:03:10,226 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 161.94.71.4 5201 < testfile
2025-07-31 05:03:10,844 - DEBUG - 1+0 records in
1+0 records out
100 bytes (100 B) copied, 0.00134623 s, 74.3 kB/s
2025-07-31 05:03:10,844 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:03:10,844 - DEBUG - ls -l testfile
2025-07-31 05:03:11,208 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:11,208 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:14,209 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:03:14,210 - DEBUG - ls -l testfile
2025-07-31 05:03:14,670 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:14,670 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:17,671 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:03:17,672 - DEBUG - ls -l testfile
2025-07-31 05:03:18,063 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:18,064 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:21,064 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:03:21,065 - DEBUG - ls -l testfile
2025-07-31 05:03:21,440 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:21,441 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:24,441 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:03:24,442 - DEBUG - ls -l testfile
2025-07-31 05:03:24,847 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:24,847 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:27,848 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:03:27,849 - DEBUG - ls -l testfile
2025-07-31 05:03:28,255 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:28,256 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:31,256 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:03:31,257 - DEBUG - ls -l testfile
2025-07-31 05:03:31,617 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:31,617 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:34,618 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:03:34,618 - DEBUG - ls -l testfile
2025-07-31 05:03:35,010 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:35,010 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:38,010 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:03:38,011 - DEBUG - ls -l testfile
2025-07-31 05:03:38,392 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:38,392 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:41,394 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:03:41,394 - DEBUG - ls -l testfile
2025-07-31 05:03:41,917 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:41,918 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:44,919 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.57, gateway password: c0ntrail123
2025-07-31 05:03:44,919 - DEBUG - ls -l testfile
2025-07-31 05:03:45,284 - DEBUG - -rw-r--r-- 1 root root 64 Jul 31 05:01 testfile
2025-07-31 05:03:45,284 - DEBUG - File size 100 verification failed on the VM, will retry after 3 seconds
2025-07-31 05:03:45,284 - INFO - File transfer verification for file size 100 failed on the VM ctest-TestMtuProbe-15541328-14386499
2025-07-31 05:03:45,287 - INFO - Deleting VM ctest-TestMtuProbe-15541328-14386499
2025-07-31 05:03:45,404 - INFO - Deleting VM ctest-TestMtuProbe-15541328-27761176
2025-07-31 05:03:45,485 - INFO - Deleting VN ctest-vn-93417578
2025-07-31 05:03:45,646 - DEBUG - VN 57f03fbd-2014-4b85-acb1-0d3ccb0e6724 still in use: Unable to complete operation on network 57f03fbd-2014-4b85-acb1-0d3ccb0e6724. There are one or more ports still in use on the network.
Neutron server returns request_ids: ['req-ff05892a-4f4f-4005-855b-2cde990de49c']
2025-07-31 05:03:45,646 - WARNING - Deleting VN ctest-vn-93417578 failed..Will retry
2025-07-31 05:03:47,910 - DEBUG - Response for deleting network ()
2025-07-31 05:03:47,910 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/3a818f26-b148-455f-8164-2a9ad49a209f
2025-07-31 05:03:47,954 - DEBUG - Response Code: 404
2025-07-31 05:03:47,954 - DEBUG - Requesting: http://10.0.0.41:8082/domains
2025-07-31 05:03:47,969 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7
2025-07-31 05:03:48,035 - DEBUG - Requesting: http://10.0.0.41:8082/project/e8822bde-b9be-4614-8a35-0e1e4b75b1f1
2025-07-31 05:03:48,142 - INFO - Validated that VN ctest-vn-93417578 is not found in API Server
2025-07-31 05:03:48,157 - DEBUG - VN ctest-vn-93417578 is not present in Agent 10.0.0.27
2025-07-31 05:03:48,175 - DEBUG - VN ctest-vn-93417578 is not present in Agent 10.0.0.57
2025-07-31 05:03:48,175 - INFO - Validated that VN ctest-vn-93417578 is not in any agent
2025-07-31 05:03:48,217 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn-93417578 info
2025-07-31 05:03:49,618 - ERROR - AssertionError
Python 3.6.8: /usr/bin/python3
Thu Jul 31 05:03:45 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={})
78 log.info('Initial checks done. Running the testcase now')
79 log.info('')
80 result = function(self, *args, **kwargs)
81 if self.inputs.upgrade:
82 pid = os.getpid()
result = None
function =
self =
args = ()
kwargs = {}
/contrail-test/tcutils/util.py in wrapper(self=, *func_args=(), **func_kwargs={})
1039 raise testtools.TestCase.skipException(msg)
1040
1041 return f(self, *func_args, **func_kwargs)
1042 return wrapper
1043 return decorator
f =
self =
func_args = ()
func_kwargs = {}
/contrail-test/scripts/vrouter/test_tcp_mtu_probe.py in test_mtu_probe_inter_node(self=)
84 self.configure_tcp_mtu_probe(client,server,value)
85 self.verify_icmp_traffic(client,server)
86 self.verify_tcp_traffic(client,server, value)
87 return True
88
self =
self.verify_tcp_traffic = >
client =
server =
value = 2
/contrail-test/scripts/vrouter/test_tcp_mtu_probe.py in verify_tcp_traffic(self=, client=, server=, value=2)
42 if value is 2:
43 sport += value
44 assert self.send_nc_traffic(client, server, sport, dport, proto), "TCP traffic failed"
45 return True
46
self =
self.send_nc_traffic = >
client =
server =
sport = 10003
dport = 5201
proto = 'tcp'
AssertionError: TCP traffic failed
__cause__ = None
__class__ =
__context__ = None
__delattr__ =
__dict__ = {}
__dir__ =
__doc__ = 'Assertion failed.'
__eq__ =
__format__ =
__ge__ =
__getattribute__ =
__gt__ =
__hash__ =
__init__ =
__init_subclass__ =
__le__ =
__lt__ =
__ne__ =
__new__ =
__reduce__ =
__reduce_ex__ =
__repr__ =
__setattr__ =
__setstate__ =
__sizeof__ =
__str__ =
__subclasshook__ =
__suppress_context__ = False
__traceback__ =
args = ('TCP traffic failed',)
with_traceback =
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
result = function(self, *args, **kwargs)
File "/contrail-test/tcutils/util.py", line 1041, in wrapper
return f(self, *func_args, **func_kwargs)
File "/contrail-test/scripts/vrouter/test_tcp_mtu_probe.py", line 86, in test_mtu_probe_inter_node
self.verify_tcp_traffic(client,server, value)
File "/contrail-test/scripts/vrouter/test_tcp_mtu_probe.py", line 44, in verify_tcp_traffic
assert self.send_nc_traffic(client, server, sport, dport, proto), "TCP traffic failed"
AssertionError: TCP traffic failed
2025-07-31 05:03:49,657 - DEBUG - No XMPP flaps were noticed during the test
2025-07-31 05:03:49,658 - INFO -
2025-07-31 05:03:49,658 - INFO - END TEST : test_mtu_probe_inter_node : FAILED[0:08:59]
2025-07-31 05:03:49,658 - INFO - --------------------------------------------------------------------------------
2025-07-31 05:03:49,674 - INFO - ================================================================================
2025-07-31 05:03:49,675 - INFO - STARTING TEST : test_mtu_probe_intra_node
2025-07-31 05:03:49,675 - INFO - TEST DESCRIPTION :
Description:
check traffic configuring different tcp_mtu_probe vlaues on vm
MTU probing values: 0, 1, 2
0 - Disabled (default)
1 - Disabled by default, Enabled only when ICMP black hole detected
2 - Always enabled, using the initial MSS value of tcp_base_mss
system internally enables bit values for point 1&2
For test purpose,
Idea is to check traffic works fine on configuring values
Vm's across compute node
steps:
1. create 1 VN and launch 2 VMs in same compute node
2. client and server VMs in same compute node
3. configure tcp_mtu_probe to 0, 1, 2
4. send icmp traffic assert if ping fails
5. send tcp traffic assert if tcp traffic fails
Pass criteria:
1. If icmp/tcp traffic work fine configuring tcp_mtu_probe values
2025-07-31 05:03:51,070 - DEBUG - Nothing to compare xmpp stats {'10.0.0.27': {'10.20.0.193': '0', '10.20.0.18': '0'}, '10.0.0.57': {'10.20.0.18': '0', '10.20.0.193': '0'}} with
2025-07-31 05:03:51,070 - INFO - Initial checks done. Running the testcase now
2025-07-31 05:03:51,070 - INFO -
2025-07-31 05:03:51,539 - DEBUG - Response for create_network : {'network': {'id': 'b16f7087-63cd-46a6-913d-359796e57d1c', 'name': 'ctest-vn-28174710', 'tenant_id': 'e8822bdeb9be46148a350e1e4b75b1f1', 'project_id': 'e8822bdeb9be46148a350e1e4b75b1f1', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestMtuProbe-15541328', 'ctest-vn-28174710'], 'port_security_enabled': True, 'description': ''}}
2025-07-31 05:03:51,792 - DEBUG - Response for create_subnet : {'subnet': {'id': 'd6902232-871e-4b7c-a862-18b2e7d6e5cb', 'name': '', 'tenant_id': 'e8822bdeb9be46148a350e1e4b75b1f1', 'network_id': 'b16f7087-63cd-46a6-913d-359796e57d1c', 'ip_version': 4, 'cidr': '5.28.8.64/26', 'allocation_pools': [{'start': '5.28.8.66', 'end': '5.28.8.126'}], 'gateway_ip': '5.28.8.65', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '5.28.8.66', 'tags': [], 'project_id': 'e8822bdeb9be46148a350e1e4b75b1f1'}}
2025-07-31 05:03:51,828 - INFO - Created VN ctest-vn-28174710
2025-07-31 05:03:51,843 - DEBUG - VN ctest-vn-28174710 UUID is b16f7087-63cd-46a6-913d-359796e57d1c
2025-07-31 05:03:51,954 - DEBUG - Requesting: http://10.0.0.41:8082/domains
2025-07-31 05:03:51,963 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7
2025-07-31 05:03:51,976 - DEBUG - Requesting: http://10.0.0.41:8082/project/e8822bde-b9be-4614-8a35-0e1e4b75b1f1
2025-07-31 05:03:52,102 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/b16f7087-63cd-46a6-913d-359796e57d1c
2025-07-31 05:03:52,120 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/b16f7087-63cd-46a6-913d-359796e57d1c
2025-07-31 05:03:52,131 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/7f9efe65-2124-42c5-88ca-39cdf2bd6428
2025-07-31 05:03:52,146 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/7f9efe65-2124-42c5-88ca-39cdf2bd6428
2025-07-31 05:03:52,160 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/9f40d27d-b14f-4fa1-961f-d002b53e2be8
2025-07-31 05:03:52,168 - DEBUG - Route Targets: ['target:64512:8000005']
2025-07-31 05:03:52,168 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/b16f7087-63cd-46a6-913d-359796e57d1c
2025-07-31 05:03:52,183 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/7f9efe65-2124-42c5-88ca-39cdf2bd6428
2025-07-31 05:03:52,205 - INFO - Verified VN network id 11 for VN b16f7087-63cd-46a6-913d-359796e57d1c
2025-07-31 05:03:52,205 - INFO - Verifications in API Server for VN ctest-vn-28174710 passed
2025-07-31 05:03:52,205 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/b16f7087-63cd-46a6-913d-359796e57d1c
2025-07-31 05:03:52,217 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/7f9efe65-2124-42c5-88ca-39cdf2bd6428
2025-07-31 05:03:52,228 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/7f9efe65-2124-42c5-88ca-39cdf2bd6428
2025-07-31 05:03:52,239 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/9f40d27d-b14f-4fa1-961f-d002b53e2be8
2025-07-31 05:03:52,254 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-15541328', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12785561593925420710', 'uuid-lslong': '10465580034266856732'}, 'enable': 'true', 'created': '2025-07-31T05:03:51', 'last-modified': '2025-07-31T05:03:51', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.563463'}
2025-07-31 05:03:52,261 - DEBUG - Route Targets: ['target:64512:8000005']
2025-07-31 05:03:52,271 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-15541328', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12785561593925420710', 'uuid-lslong': '10465580034266856732'}, 'enable': 'true', 'created': '2025-07-31T05:03:51', 'last-modified': '2025-07-31T05:03:51', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.581138'}
2025-07-31 05:03:52,278 - DEBUG - Route Targets: ['target:64512:8000005']
2025-07-31 05:03:52,286 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestMtuProbe-15541328', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12785561593925420710', 'uuid-lslong': '10465580034266856732'}, 'enable': 'true', 'created': '2025-07-31T05:03:51', 'last-modified': '2025-07-31T05:03:51', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.599730'}
2025-07-31 05:03:52,293 - DEBUG - Route Targets: ['target:64512:8000005']
2025-07-31 05:03:52,294 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn-28174710 passed
2025-07-31 05:03:52,294 - DEBUG - ====Verifying policy data for ctest-vn-28174710 in API_Server ======
2025-07-31 05:03:52,294 - DEBUG - Requesting: http://10.0.0.41:8082/domains
2025-07-31 05:03:52,302 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7
2025-07-31 05:03:52,322 - DEBUG - Requesting: http://10.0.0.41:8082/project/e8822bde-b9be-4614-8a35-0e1e4b75b1f1
2025-07-31 05:03:52,468 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/b16f7087-63cd-46a6-913d-359796e57d1c
2025-07-31 05:03:52,482 - DEBUG - =>VN ctest-vn-28174710 has no policy to be verified
2025-07-31 05:03:52,482 - DEBUG - Verifying the vn in opserver
2025-07-31 05:03:52,482 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 virtual network link through opserver 10.0.0.41
2025-07-31 05:03:52,482 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks
2025-07-31 05:03:52,568 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710?flat'}
2025-07-31 05:03:52,568 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 virtual network link through opserver 10.0.0.38
2025-07-31 05:03:52,568 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks
2025-07-31 05:03:52,668 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710?flat'}
2025-07-31 05:03:52,668 - DEBUG - Verifying the default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 virtual network link through opserver 10.0.0.61
2025-07-31 05:03:52,668 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks
2025-07-31 05:03:52,781 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710?flat'}
2025-07-31 05:03:52,782 - INFO - Validated that VN default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is found in opserver
2025-07-31 05:03:52,839 - DEBUG - Do not have enough data to verify VN in agent
2025-07-31 05:03:52,855 - DEBUG - VRF ids for VN ctest-vn-28174710: {}
2025-07-31 05:03:52,856 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']}
2025-07-31 05:03:52,856 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']}
2025-07-31 05:03:54,827 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-1)
2025-07-31 05:03:54,827 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3732-1', 'an-jenkins-deploy-platform-ansible-os-3732-2']}
2025-07-31 05:03:56,402 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-1)
2025-07-31 05:03:56,402 - INFO - Waiting for VM ctest-TestMtuProbe-15541328-22439080 to be up..
2025-07-31 05:03:56,509 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2025-07-31 05:04:01,607 - DEBUG - VM is in ACTIVE state now
2025-07-31 05:04:01,607 - INFO - VM name : ctest-TestMtuProbe-15541328-22439080
2025-07-31 05:04:01,712 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 ID is 0ea4c3b4-c17f-4dce-959b-178975d4f446
2025-07-31 05:04:01,712 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 launched on Node an-jenkins-deploy-platform-ansible-os-3732-1
2025-07-31 05:04:01,804 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/0ea4c3b4-c17f-4dce-959b-178975d4f446
2025-07-31 05:04:01,838 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/b6eb1f5d-5a71-4b22-bb17-56c1c38c373b
2025-07-31 05:04:05,100 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1024ms')
2025-07-31 05:04:05,101 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed!
2025-07-31 05:04:05,128 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set
2025-07-31 05:04:09,231 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1030ms')
2025-07-31 05:04:09,231 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed!
2025-07-31 05:04:09,262 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set
2025-07-31 05:04:13,352 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1024ms')
2025-07-31 05:04:13,353 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed!
2025-07-31 05:04:13,379 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set
2025-07-31 05:04:17,451 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1001ms')
2025-07-31 05:04:17,451 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed!
2025-07-31 05:04:17,479 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set
2025-07-31 05:04:21,549 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1002ms')
2025-07-31 05:04:21,549 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed!
2025-07-31 05:04:21,566 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set
2025-07-31 05:04:25,649 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms')
2025-07-31 05:04:25,649 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed!
2025-07-31 05:04:25,682 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set
2025-07-31 05:04:29,773 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms')
2025-07-31 05:04:29,773 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed!
2025-07-31 05:04:29,799 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set
2025-07-31 05:04:33,862 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1002ms')
2025-07-31 05:04:33,863 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed!
2025-07-31 05:04:33,889 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set
2025-07-31 05:04:37,967 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms')
2025-07-31 05:04:37,967 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed!
2025-07-31 05:04:37,984 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set
2025-07-31 05:04:42,058 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms')
2025-07-31 05:04:42,059 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed!
2025-07-31 05:04:42,094 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set
2025-07-31 05:04:46,193 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms')
2025-07-31 05:04:46,193 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 failed!
2025-07-31 05:04:46,222 - DEBUG - Gateway for vn default-domain:ctest-TestMtuProbe-15541328:ctest-vn-28174710 is 5.28.8.65 and allocation pool is NOT set
2025-07-31 05:04:48,296 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=6.74 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=3.83 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 3.825/5.281/6.737/1.456 ms')
2025-07-31 05:04:48,296 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-TestMtuProbe-15541328-22439080 passed
2025-07-31 05:04:48,419 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:04:48,419 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-22439080, IP 5.28.8.67, Port 22
2025-07-31 05:04:48,488 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-07-31 05:04:48,650 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 is NOT ready for SSH connections, VM status: ACTIVE
2025-07-31 05:04:53,651 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:04:53,652 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-22439080, IP 5.28.8.67, Port 22
2025-07-31 05:04:53,723 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-07-31 05:04:53,910 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 is NOT ready for SSH connections, VM status: ACTIVE
2025-07-31 05:04:58,911 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:04:58,912 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-22439080, IP 5.28.8.67, Port 22
2025-07-31 05:04:59,004 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-07-31 05:04:59,144 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 is NOT ready for SSH connections, VM status: ACTIVE
2025-07-31 05:05:04,145 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:05:04,145 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-22439080, IP 5.28.8.67, Port 22
2025-07-31 05:05:04,219 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-07-31 05:05:04,373 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 is NOT ready for SSH connections, VM status: ACTIVE
2025-07-31 05:05:09,374 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:05:09,374 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-22439080, IP 5.28.8.67, Port 22
2025-07-31 05:05:09,447 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-07-31 05:05:09,654 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 is NOT ready for SSH connections, VM status: ACTIVE
2025-07-31 05:05:14,655 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:05:14,655 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-22439080, IP 5.28.8.67, Port 22
2025-07-31 05:05:14,817 - DEBUG - VM ctest-TestMtuProbe-15541328-22439080 is ready for SSH connections
2025-07-31 05:05:14,817 - INFO - Waiting for VM ctest-TestMtuProbe-15541328-05529872 to be up..
2025-07-31 05:05:15,021 - DEBUG - VM is in ACTIVE state now
2025-07-31 05:05:15,021 - INFO - VM name : ctest-TestMtuProbe-15541328-05529872
2025-07-31 05:05:15,183 - DEBUG - VM ctest-TestMtuProbe-15541328-05529872 ID is 85fb464b-3624-40a6-95ec-b2d247f7bc67
2025-07-31 05:05:15,183 - DEBUG - VM ctest-TestMtuProbe-15541328-05529872 launched on Node an-jenkins-deploy-platform-ansible-os-3732-1
2025-07-31 05:05:15,351 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/85fb464b-3624-40a6-95ec-b2d247f7bc67
2025-07-31 05:05:15,378 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/611939da-1dad-453f-99bf-4814ab89ffe0
2025-07-31 05:05:16,645 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=4.35 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=1.19 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.193/2.769/4.345/1.576 ms')
2025-07-31 05:05:16,646 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-TestMtuProbe-15541328-05529872 passed
2025-07-31 05:05:16,795 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:05:16,795 - DEBUG - Waiting to SSH to VM ctest-TestMtuProbe-15541328-05529872, IP 5.28.8.68, Port 22
2025-07-31 05:05:16,985 - DEBUG - VM ctest-TestMtuProbe-15541328-05529872 is ready for SSH connections
2025-07-31 05:05:16,985 - DEBUG - Running remote_cmd, Cmd : echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:05:16,985 - DEBUG - echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing
2025-07-31 05:06:11,793 - DEBUG - None
2025-07-31 05:06:11,794 - DEBUG - Running remote_cmd, Cmd : echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:06:11,794 - DEBUG - echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing
2025-07-31 05:06:54,017 - DEBUG - None
2025-07-31 05:06:54,018 - INFO - {'echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing': None}
2025-07-31 05:06:54,018 - INFO - {'echo 0 > /proc/sys/net/ipv4/tcp_mtu_probing': None}
2025-07-31 05:06:54,020 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:06:54,020 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 5.28.8.68, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:06:54,020 - DEBUG - ping -s 56 -c 3 -W 1 5.28.8.68
2025-07-31 05:06:56,604 - DEBUG - PING 5.28.8.68 (5.28.8.68) 56(84) bytes of data.
64 bytes from 5.28.8.68: icmp_req=1 ttl=64 time=8.17 ms
64 bytes from 5.28.8.68: icmp_req=2 ttl=64 time=1.01 ms
64 bytes from 5.28.8.68: icmp_req=3 ttl=64 time=1.31 ms
--- 5.28.8.68 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.012/3.499/8.170/3.305 ms
2025-07-31 05:06:56,605 - INFO - Ping to IP 5.28.8.68 from VM ctest-TestMtuProbe-15541328-22439080 passed
2025-07-31 05:06:56,605 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:06:56,606 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:06:56,606 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile &
2025-07-31 05:06:57,199 - DEBUG - total 132
drwxr-xr-x 11 ubuntu ubuntu 4096 Dec 29 2017 .
drwxr-xr-x 3 root root 4096 Jan 4 2013 ..
-rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history
-rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout
-rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc
drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache
-rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh
drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig
drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm
drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins
-rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst
drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla
-rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh
-rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr
drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh
-rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic
drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0
-rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz
-rw------- 1 root root 4695 Dec 17 2017 .viminfo
-rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority
-rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors
2025-07-31 05:06:57,199 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:06:57,199 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile &
2025-07-31 05:06:57,647 - DEBUG - None
2025-07-31 05:06:57,648 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 5.28.8.68 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:06:57,648 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10001 5.28.8.68 5201 < testfile
2025-07-31 05:07:00,320 - DEBUG - 1+0 records in
1+0 records out
100 bytes (100 B) copied, 0.00107571 s, 93.0 kB/s
2025-07-31 05:07:00,320 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:00,320 - DEBUG - ls -l testfile
2025-07-31 05:07:00,686 - DEBUG - -rw-r--r-- 1 root root 100 Jul 31 05:06 testfile
2025-07-31 05:07:00,686 - INFO - File transfer verification for file size 100 passed on the VM ctest-TestMtuProbe-15541328-05529872
2025-07-31 05:07:00,686 - DEBUG - Running remote_cmd, Cmd : echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:00,686 - DEBUG - echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing
2025-07-31 05:07:01,150 - DEBUG - None
2025-07-31 05:07:01,150 - DEBUG - Running remote_cmd, Cmd : echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:01,150 - DEBUG - echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing
2025-07-31 05:07:01,588 - DEBUG - None
2025-07-31 05:07:01,588 - INFO - {'echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing': None}
2025-07-31 05:07:01,588 - INFO - {'echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing': None}
2025-07-31 05:07:01,588 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:07:01,588 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 5.28.8.68, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:01,588 - DEBUG - ping -s 56 -c 3 -W 1 5.28.8.68
2025-07-31 05:07:04,141 - DEBUG - PING 5.28.8.68 (5.28.8.68) 56(84) bytes of data.
64 bytes from 5.28.8.68: icmp_req=1 ttl=64 time=2.28 ms
64 bytes from 5.28.8.68: icmp_req=2 ttl=64 time=0.881 ms
64 bytes from 5.28.8.68: icmp_req=3 ttl=64 time=0.836 ms
--- 5.28.8.68 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.836/1.334/2.286/0.673 ms
2025-07-31 05:07:04,141 - INFO - Ping to IP 5.28.8.68 from VM ctest-TestMtuProbe-15541328-22439080 passed
2025-07-31 05:07:04,141 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:07:04,141 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:04,141 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile &
2025-07-31 05:07:04,774 - DEBUG - total 132
drwxr-xr-x 11 ubuntu ubuntu 4096 Jul 31 05:07 .
drwxr-xr-x 3 root root 4096 Jan 4 2013 ..
-rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history
-rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout
-rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc
drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache
-rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh
drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig
drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm
drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins
-rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst
drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla
-rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh
-rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr
drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh
-rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic
drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0
-rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz
-rw------- 1 root root 4695 Dec 17 2017 .viminfo
-rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority
-rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors
2025-07-31 05:07:04,775 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:04,775 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile &
2025-07-31 05:07:05,238 - DEBUG - None
2025-07-31 05:07:05,239 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10002 5.28.8.68 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:05,239 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10002 5.28.8.68 5201 < testfile
2025-07-31 05:07:07,837 - DEBUG - 1+0 records in
1+0 records out
100 bytes (100 B) copied, 0.00117167 s, 85.3 kB/s
2025-07-31 05:07:07,838 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:07,838 - DEBUG - ls -l testfile
2025-07-31 05:07:08,180 - DEBUG - -rw-r--r-- 1 root root 100 Jul 31 05:07 testfile
2025-07-31 05:07:08,180 - INFO - File transfer verification for file size 100 passed on the VM ctest-TestMtuProbe-15541328-05529872
2025-07-31 05:07:08,180 - DEBUG - Running remote_cmd, Cmd : echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:08,181 - DEBUG - echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing
2025-07-31 05:07:08,616 - DEBUG - None
2025-07-31 05:07:08,617 - DEBUG - Running remote_cmd, Cmd : echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:08,617 - DEBUG - echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing
2025-07-31 05:07:09,092 - DEBUG - None
2025-07-31 05:07:09,093 - INFO - {'echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing': None}
2025-07-31 05:07:09,093 - INFO - {'echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing': None}
2025-07-31 05:07:09,093 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:07:09,093 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 5.28.8.68, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:09,093 - DEBUG - ping -s 56 -c 3 -W 1 5.28.8.68
2025-07-31 05:07:11,570 - DEBUG - PING 5.28.8.68 (5.28.8.68) 56(84) bytes of data.
64 bytes from 5.28.8.68: icmp_req=1 ttl=64 time=3.05 ms
64 bytes from 5.28.8.68: icmp_req=2 ttl=64 time=1.10 ms
64 bytes from 5.28.8.68: icmp_req=3 ttl=64 time=1.13 ms
--- 5.28.8.68 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2005ms
rtt min/avg/max/mdev = 1.101/1.762/3.053/0.912 ms
2025-07-31 05:07:11,570 - INFO - Ping to IP 5.28.8.68 from VM ctest-TestMtuProbe-15541328-22439080 passed
2025-07-31 05:07:11,571 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-07-31 05:07:11,571 - DEBUG - Running remote_cmd, Cmd : rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:11,571 - DEBUG - nohup rm -f testfile;ls -la;nc -4 -q 2 -w 5 -ll -p 5201 > testfile &
2025-07-31 05:07:12,153 - DEBUG - total 132
drwxr-xr-x 11 ubuntu ubuntu 4096 Jul 31 05:07 .
drwxr-xr-x 3 root root 4096 Jan 4 2013 ..
-rw------- 1 ubuntu ubuntu 2689 Dec 29 2017 .bash_history
-rw-r--r-- 1 ubuntu ubuntu 220 Apr 3 2012 .bash_logout
-rw-r--r-- 1 ubuntu ubuntu 3486 Apr 3 2012 .bashrc
drwx------ 2 ubuntu ubuntu 4096 Jan 9 2013 .cache
-rwxrwxr-x 1 ubuntu ubuntu 1431 Feb 5 2015 flow_test_pktgen.sh
drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fontconfig
drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 9 2013 .fvwm
drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 29 2016 .jenkins
-rw------- 1 ubuntu ubuntu 52 Jul 2 2014 .lesshst
drwx------ 4 ubuntu ubuntu 4096 Jan 9 2013 .mozilla
-rwxrwxr-x 1 ubuntu ubuntu 1378 Jun 27 2014 pktgen_new.sh
-rw-r--r-- 1 ubuntu ubuntu 675 Apr 3 2012 .profile
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 scr
drwx------ 2 ubuntu ubuntu 4096 Feb 3 2015 .ssh
-rw-r--r-- 1 ubuntu ubuntu 0 Jan 9 2013 .sudo_as_admin_successful
drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 2014 traffic
drwxr-xr-x 4 root root 4096 Jun 27 2014 traffic-1.0
-rw-rw-r-- 1 ubuntu ubuntu 16361 Jan 9 2015 traffic-1.0.tar.gz
-rw------- 1 root root 4695 Dec 17 2017 .viminfo
-rw------- 1 ubuntu ubuntu 152 Jul 2 2014 .Xauthority
-rw------- 1 ubuntu ubuntu 29828 Jan 9 2013 .xsession-errors
2025-07-31 05:07:12,153 - DEBUG - Running remote_cmd, Cmd : nc -4 -q 2 -w 5 -ll 5201 > testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:12,153 - DEBUG - nohup nc -4 -q 2 -w 5 -ll 5201 > testfile &
2025-07-31 05:07:12,667 - DEBUG - None
2025-07-31 05:07:12,668 - DEBUG - Running remote_cmd, Cmd : dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 5.28.8.68 5201 < testfile, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:12,668 - DEBUG - dd bs=100 count=1 if=/dev/zero of=testfile;nc -4 -q 2 -w 5 -p 10003 5.28.8.68 5201 < testfile
2025-07-31 05:07:15,240 - DEBUG - 1+0 records in
1+0 records out
100 bytes (100 B) copied, 0.000948353 s, 105 kB/s
2025-07-31 05:07:15,240 - DEBUG - Running remote_cmd, Cmd : ls -l testfile, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-07-31 05:07:15,241 - DEBUG - ls -l testfile
2025-07-31 05:07:15,576 - DEBUG - -rw-r--r-- 1 root root 100 Jul 31 05:07 testfile
2025-07-31 05:07:15,577 - INFO - File transfer verification for file size 100 passed on the VM ctest-TestMtuProbe-15541328-05529872
2025-07-31 05:07:15,577 - INFO - Deleting VM ctest-TestMtuProbe-15541328-05529872
2025-07-31 05:07:15,713 - INFO - Deleting VM ctest-TestMtuProbe-15541328-22439080
2025-07-31 05:07:15,784 - INFO - Deleting VN ctest-vn-28174710
2025-07-31 05:07:15,836 - DEBUG - VN b16f7087-63cd-46a6-913d-359796e57d1c still in use: Unable to complete operation on network b16f7087-63cd-46a6-913d-359796e57d1c. There are one or more ports still in use on the network.
Neutron server returns request_ids: ['req-ff315fdc-ac8f-40cf-bf91-f03935cdf27e']
2025-07-31 05:07:15,836 - WARNING - Deleting VN ctest-vn-28174710 failed..Will retry
2025-07-31 05:07:18,024 - DEBUG - Response for deleting network ()
2025-07-31 05:07:18,024 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/7f9efe65-2124-42c5-88ca-39cdf2bd6428
2025-07-31 05:07:18,033 - DEBUG - Response Code: 404
2025-07-31 05:07:18,033 - DEBUG - Requesting: http://10.0.0.41:8082/domains
2025-07-31 05:07:18,041 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7
2025-07-31 05:07:18,069 - DEBUG - Requesting: http://10.0.0.41:8082/project/e8822bde-b9be-4614-8a35-0e1e4b75b1f1
2025-07-31 05:07:18,169 - INFO - Validated that VN ctest-vn-28174710 is not found in API Server
2025-07-31 05:07:18,184 - DEBUG - VN ctest-vn-28174710 is not present in Agent 10.0.0.27
2025-07-31 05:07:18,199 - DEBUG - VN ctest-vn-28174710 is not present in Agent 10.0.0.57
2025-07-31 05:07:18,199 - INFO - Validated that VN ctest-vn-28174710 is not in any agent
2025-07-31 05:07:18,251 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn-28174710 info
2025-07-31 05:07:19,598 - DEBUG - No XMPP flaps were noticed during the test
2025-07-31 05:07:19,598 - INFO - END TEST : test_mtu_probe_intra_node : PASSED[0:03:30]
2025-07-31 05:07:19,598 - INFO - --------------------------------------------------------------------------------
2025-07-31 05:07:20,483 - INFO - Deleted project: ctest-TestMtuProbe-15541328, ID : e8822bde-b9be-4614-8a35-0e1e4b75b1f1