2025-12-11 04:11:45,891 - INFO - Domain Default found not creating 2025-12-11 04:11:46,066 - INFO - Project ctest-TestBasicVMVN0-89570916 not found, creating it 2025-12-11 04:11:46,587 - INFO - Created Project:ctest-TestBasicVMVN0-89570916, ID : 2bc25706-3e1b-4dcd-bbbb-0261d020b882 2025-12-11 04:11:48,066 - INFO - ================================================================================ 2025-12-11 04:11:48,067 - INFO - STARTING TEST : test_control_node_switchover 2025-12-11 04:11:48,067 - INFO - TEST DESCRIPTION : Stop the control node and check peering with agent fallback to other control node. 1. Pick one VN from respource pool which has 2 VM's in it 2. Verify ping between VM's 3. Find active control node in cluster by agent inspect 4. Stop control service on active control node 5. Verify agents are connected to new active control-node using xmpp connections 6. Bring back control service on previous active node 7. Verify ping between VM's again after bringing up control serveice Pass criteria: Step 2,5 and 7 should pass 2025-12-11 04:11:49,315 - DEBUG - Skipping xmpp flap check 2025-12-11 04:11:49,315 - INFO - Initial checks done. Running the testcase now 2025-12-11 04:11:49,316 - INFO - 2025-12-11 04:11:49,978 - DEBUG - Response for create_network : {'network': {'id': '3cefc596-4b06-4e66-8c15-44705d46adb5', 'name': 'ctest-vn1-31264405', 'tenant_id': '2bc257063e1b4dcdbbbb0261d020b882', 'project_id': '2bc257063e1b4dcdbbbb0261d020b882', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicVMVN0-89570916', 'ctest-vn1-31264405'], 'port_security_enabled': True, 'description': ''}} 2025-12-11 04:11:50,245 - DEBUG - Response for create_subnet : {'subnet': {'id': '3df3b007-613b-4a25-b0fd-01f465aebbe2', 'name': '', 'tenant_id': '2bc257063e1b4dcdbbbb0261d020b882', 'network_id': '3cefc596-4b06-4e66-8c15-44705d46adb5', 'ip_version': 4, 'cidr': '192.168.1.0/24', 'allocation_pools': [{'start': '192.168.1.2', 'end': '192.168.1.254'}], 'gateway_ip': '192.168.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '192.168.1.2', 'tags': [], 'project_id': '2bc257063e1b4dcdbbbb0261d020b882'}} 2025-12-11 04:11:50,264 - INFO - Created VN ctest-vn1-31264405 2025-12-11 04:11:50,273 - DEBUG - VN ctest-vn1-31264405 UUID is 3cefc596-4b06-4e66-8c15-44705d46adb5 2025-12-11 04:11:50,337 - DEBUG - Requesting: http://10.0.0.61:8082/domains 2025-12-11 04:11:50,630 - DEBUG - Requesting: http://10.0.0.61:8082/domains 2025-12-11 04:11:50,661 - DEBUG - Requesting: http://10.0.0.61:8082/domain/35136ed0-c5bb-4823-8f5c-ae1b4c17652e 2025-12-11 04:11:50,676 - DEBUG - Requesting: http://10.0.0.61:8082/project/2bc25706-3e1b-4dcd-bbbb-0261d020b882 2025-12-11 04:11:50,780 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/3cefc596-4b06-4e66-8c15-44705d46adb5 2025-12-11 04:11:50,795 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/3cefc596-4b06-4e66-8c15-44705d46adb5 2025-12-11 04:11:50,804 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/3664c295-184e-47d1-9c3d-bfdf79b26b1d 2025-12-11 04:11:50,813 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/3664c295-184e-47d1-9c3d-bfdf79b26b1d 2025-12-11 04:11:50,820 - DEBUG - Requesting: http://10.0.0.61:8082/route-target/65048d26-5d88-4ef3-82f9-99fd57f21701 2025-12-11 04:11:50,827 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-11 04:11:50,827 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/3cefc596-4b06-4e66-8c15-44705d46adb5 2025-12-11 04:11:50,837 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/3664c295-184e-47d1-9c3d-bfdf79b26b1d 2025-12-11 04:11:50,853 - INFO - Verified VN network id 14 for VN 3cefc596-4b06-4e66-8c15-44705d46adb5 2025-12-11 04:11:50,854 - INFO - Verifications in API Server for VN ctest-vn1-31264405 passed 2025-12-11 04:11:50,854 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/3cefc596-4b06-4e66-8c15-44705d46adb5 2025-12-11 04:11:50,865 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/3664c295-184e-47d1-9c3d-bfdf79b26b1d 2025-12-11 04:11:50,872 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/3664c295-184e-47d1-9c3d-bfdf79b26b1d 2025-12-11 04:11:50,879 - DEBUG - Requesting: http://10.0.0.61:8082/route-target/65048d26-5d88-4ef3-82f9-99fd57f21701 2025-12-11 04:11:50,892 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-89570916', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4390945411003993702', 'uuid-lslong': '10094049389212773813'}, 'enable': 'true', 'created': '2025-12-11T04:11:49', 'last-modified': '2025-12-11T04:11:50', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.695788'} 2025-12-11 04:11:50,898 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-11 04:11:50,905 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-89570916', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4390945411003993702', 'uuid-lslong': '10094049389212773813'}, 'enable': 'true', 'created': '2025-12-11T04:11:49', 'last-modified': '2025-12-11T04:11:50', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.700674'} 2025-12-11 04:11:50,912 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-11 04:11:50,917 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-89570916', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4390945411003993702', 'uuid-lslong': '10094049389212773813'}, 'enable': 'true', 'created': '2025-12-11T04:11:49', 'last-modified': '2025-12-11T04:11:50', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.724303'} 2025-12-11 04:11:50,923 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-11 04:11:50,923 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-31264405 passed 2025-12-11 04:11:50,923 - DEBUG - ====Verifying policy data for ctest-vn1-31264405 in API_Server ====== 2025-12-11 04:11:50,923 - DEBUG - Requesting: http://10.0.0.61:8082/domains 2025-12-11 04:11:50,929 - DEBUG - Requesting: http://10.0.0.61:8082/domain/35136ed0-c5bb-4823-8f5c-ae1b4c17652e 2025-12-11 04:11:50,943 - DEBUG - Requesting: http://10.0.0.61:8082/project/2bc25706-3e1b-4dcd-bbbb-0261d020b882 2025-12-11 04:11:51,043 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/3cefc596-4b06-4e66-8c15-44705d46adb5 2025-12-11 04:11:51,053 - DEBUG - =>VN ctest-vn1-31264405 has no policy to be verified 2025-12-11 04:11:51,053 - DEBUG - Verifying the vn in opserver 2025-12-11 04:11:51,053 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 virtual network link through opserver 10.0.0.61 2025-12-11 04:11:51,053 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-12-11 04:11:51,354 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-12-11 04:11:51,419 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405?flat'} 2025-12-11 04:11:51,420 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 virtual network link through opserver 10.0.0.33 2025-12-11 04:11:51,420 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-12-11 04:11:51,721 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-12-11 04:11:51,776 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405?flat'} 2025-12-11 04:11:51,776 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 virtual network link through opserver 10.0.0.27 2025-12-11 04:11:51,776 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-12-11 04:11:52,072 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-12-11 04:11:52,140 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405?flat'} 2025-12-11 04:11:52,140 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is found in opserver 2025-12-11 04:11:52,161 - DEBUG - Do not have enough data to verify VN in agent 2025-12-11 04:11:52,171 - DEBUG - VRF ids for VN ctest-vn1-31264405: {} 2025-12-11 04:11:52,241 - DEBUG - Services list from nova: [, , , ] 2025-12-11 04:11:53,299 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4883-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4883-2) 2025-12-11 04:11:54,658 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4883-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4883-1) 2025-12-11 04:11:54,658 - INFO - Waiting for VM ctest-vn1_vm1-57438079 to be up.. 2025-12-11 04:11:54,743 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-12-11 04:11:59,856 - DEBUG - VM is in ACTIVE state now 2025-12-11 04:11:59,856 - INFO - VM name : ctest-vn1_vm1-57438079 2025-12-11 04:11:59,948 - DEBUG - VM ctest-vn1_vm1-57438079 ID is 034da2e8-ef2b-4480-a910-a102aa142f77 2025-12-11 04:11:59,971 - DEBUG - VM ctest-vn1_vm1-57438079 launched on Node an-jenkins-deploy-platform-ansible-os-4883-2 2025-12-11 04:12:00,074 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77 2025-12-11 04:12:00,377 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77 2025-12-11 04:12:00,417 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/3e8c2d36-9ecd-4f00-96c7-1066028279dc 2025-12-11 04:12:03,614 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1030ms') 2025-12-11 04:12:03,614 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-57438079 failed! 2025-12-11 04:12:03,627 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is 192.168.1.1 and allocation pool is NOT set 2025-12-11 04:12:07,710 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-12-11 04:12:07,710 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-57438079 failed! 2025-12-11 04:12:07,724 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is 192.168.1.1 and allocation pool is NOT set 2025-12-11 04:12:11,800 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-12-11 04:12:11,800 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-57438079 failed! 2025-12-11 04:12:11,817 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is 192.168.1.1 and allocation pool is NOT set 2025-12-11 04:12:15,893 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-12-11 04:12:15,894 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-57438079 failed! 2025-12-11 04:12:15,906 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is 192.168.1.1 and allocation pool is NOT set 2025-12-11 04:12:19,993 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms') 2025-12-11 04:12:19,994 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-57438079 failed! 2025-12-11 04:12:20,007 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is 192.168.1.1 and allocation pool is NOT set 2025-12-11 04:12:24,092 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-12-11 04:12:24,092 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-57438079 failed! 2025-12-11 04:12:24,104 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is 192.168.1.1 and allocation pool is NOT set 2025-12-11 04:12:28,192 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-12-11 04:12:28,192 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-57438079 failed! 2025-12-11 04:12:28,205 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is 192.168.1.1 and allocation pool is NOT set 2025-12-11 04:12:32,286 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms') 2025-12-11 04:12:32,286 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-57438079 failed! 2025-12-11 04:12:32,299 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is 192.168.1.1 and allocation pool is NOT set 2025-12-11 04:12:34,363 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=7.80 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.09 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.090/4.445/7.801/3.355 ms') 2025-12-11 04:12:34,363 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-57438079 passed 2025-12-11 04:12:34,432 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:12:34,432 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-57438079, IP 192.168.1.3, Port 22 2025-12-11 04:12:34,497 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-11 04:12:34,580 - DEBUG - VM ctest-vn1_vm1-57438079 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-11 04:12:39,580 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:12:39,580 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-57438079, IP 192.168.1.3, Port 22 2025-12-11 04:12:39,649 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-11 04:12:39,728 - DEBUG - VM ctest-vn1_vm1-57438079 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-11 04:12:44,729 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:12:44,729 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-57438079, IP 192.168.1.3, Port 22 2025-12-11 04:12:44,797 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-11 04:12:44,941 - DEBUG - VM ctest-vn1_vm1-57438079 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-11 04:12:49,942 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:12:49,942 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-57438079, IP 192.168.1.3, Port 22 2025-12-11 04:12:49,999 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-11 04:12:50,074 - DEBUG - VM ctest-vn1_vm1-57438079 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-11 04:12:55,074 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:12:55,075 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-57438079, IP 192.168.1.3, Port 22 2025-12-11 04:12:55,242 - DEBUG - VM ctest-vn1_vm1-57438079 is ready for SSH connections 2025-12-11 04:12:55,242 - INFO - Waiting for VM ctest-vn1_vm2-51220443 to be up.. 2025-12-11 04:12:55,321 - DEBUG - VM is in ACTIVE state now 2025-12-11 04:12:55,321 - INFO - VM name : ctest-vn1_vm2-51220443 2025-12-11 04:12:55,403 - DEBUG - VM ctest-vn1_vm2-51220443 ID is 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 2025-12-11 04:12:55,403 - DEBUG - VM ctest-vn1_vm2-51220443 launched on Node an-jenkins-deploy-platform-ansible-os-4883-1 2025-12-11 04:12:55,480 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 2025-12-11 04:12:55,489 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/28a11ee5-1428-45cc-9116-03dd0e8f7a63 2025-12-11 04:12:56,649 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.66 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.63 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1000ms\r\nrtt min/avg/max/mdev = 1.632/3.644/5.657/2.012 ms') 2025-12-11 04:12:56,649 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-51220443 passed 2025-12-11 04:12:56,716 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:12:56,717 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-51220443, IP 192.168.1.4, Port 22 2025-12-11 04:12:56,885 - DEBUG - VM ctest-vn1_vm2-51220443 is ready for SSH connections 2025-12-11 04:12:56,885 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:12:56,885 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.23, gateway password: c0ntrail123 2025-12-11 04:12:56,885 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-12-11 04:13:18,765 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=11.8 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.51 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.22 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 1.229/4.874/11.879/4.954 ms 2025-12-11 04:13:18,765 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-57438079 passed 2025-12-11 04:13:18,765 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:13:18,766 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.141, gateway password: c0ntrail123 2025-12-11 04:13:18,766 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-12-11 04:14:01,377 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=8.44 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.16 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.13 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.136/3.581/8.449/3.442 ms 2025-12-11 04:14:01,377 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-51220443 passed 2025-12-11 04:14:01,385 - INFO - Active control node from the Agent 10.0.0.23 is 10.0.0.27 2025-12-11 04:14:01,385 - INFO - Stoping the Control service in 10.0.0.27 2025-12-11 04:14:01,801 - INFO - 10.0.0.27 2025-12-11 04:14:01,801 - INFO - control:inactive 2025-12-11 04:14:01,801 - INFO - Contrail services control are down on nodes 10.0.0.27 2025-12-11 04:14:06,808 - INFO - Active control node from the Agent 10.0.0.23 is 10.0.0.61 2025-12-11 04:14:06,808 - INFO - Starting the Control service in 10.0.0.27 2025-12-11 04:14:07,110 - INFO - 10.0.0.27 2025-12-11 04:14:07,114 - INFO - control:initializing 2025-12-11 04:14:07,115 - DEBUG - defaultdict(, {'10.0.0.27': {'control': {'status': 'initializing', 'description': None}}}) 2025-12-11 04:14:07,115 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-11 04:14:12,202 - INFO - 10.0.0.27 2025-12-11 04:14:12,207 - INFO - control:active 2025-12-11 04:14:12,207 - INFO - Contrail services ['control'] are up on nodes 10.0.0.27 2025-12-11 04:14:17,307 - DEBUG - VM is in ACTIVE state now 2025-12-11 04:14:17,307 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:14:17,307 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77 2025-12-11 04:14:17,316 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/3e8c2d36-9ecd-4f00-96c7-1066028279dc 2025-12-11 04:14:17,326 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77 2025-12-11 04:14:17,334 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/3e8c2d36-9ecd-4f00-96c7-1066028279dc 2025-12-11 04:14:17,343 - DEBUG - Requesting: http://10.0.0.27:8082/instance-ip/264b0c98-a43c-448a-a3c4-e1917482e171 2025-12-11 04:14:17,353 - DEBUG - Verifying in api server 10.0.0.27 2025-12-11 04:14:17,353 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77 2025-12-11 04:14:17,360 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/3e8c2d36-9ecd-4f00-96c7-1066028279dc 2025-12-11 04:14:17,369 - DEBUG - Requesting: http://10.0.0.27:8082/instance-ip/264b0c98-a43c-448a-a3c4-e1917482e171 2025-12-11 04:14:17,377 - INFO - VM ctest-vn1_vm1-57438079 verfication in all API Servers passed 2025-12-11 04:14:17,396 - DEBUG - VM ctest-vn1_vm1-57438079 Tap interface: {'index': '3', 'name': 'tap3e8c2d36-9e', 'uuid': '3e8c2d36-9ecd-4f00-96c7-1066028279dc', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405:ctest-vn1-31264405', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405', 'vm_uuid': '034da2e8-ef2b-4480-a910-a102aa142f77', 'vm_name': 'ctest-vn1_vm1-57438079', 'ip_addr': '192.168.1.3', 'mac_addr': '02:3e:8c:2d:36:9e', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '161', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-89570916:3e8c2d36-9ecd-4f00-96c7-1066028279dc', 'sg_uuid_list': ['2b857d45-cecf-440b-a7cf-436d4e16a1fb'], 'static_route_list': None, 'vm_project_uuid': '2bc25706-3e1b-4dcd-bbbb-0261d020b882', 'admin_state': 'Enabled', 'flow_key_idx': '27', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-11 04:14:17,396 - DEBUG - Agent 10.0.0.23 vrf name: default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405:ctest-vn1-31264405 2025-12-11 04:14:17,401 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405:ctest-vn1-31264405', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405', 'table_label': '-1', 'vxlan_id': '14', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.241:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-12-11 04:14:17,484 - DEBUG - VM ctest-vn1_vm1-57438079 : Tap interface tap3e8c2d36-9e is set to Active 2025-12-11 04:14:17,484 - DEBUG - Tap interface tap3e8c2d36-9e detail : {'index': '3', 'name': 'tap3e8c2d36-9e', 'uuid': '3e8c2d36-9ecd-4f00-96c7-1066028279dc', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405:ctest-vn1-31264405', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405', 'vm_uuid': '034da2e8-ef2b-4480-a910-a102aa142f77', 'vm_name': 'ctest-vn1_vm1-57438079', 'ip_addr': '192.168.1.3', 'mac_addr': '02:3e:8c:2d:36:9e', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '161', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-89570916:3e8c2d36-9ecd-4f00-96c7-1066028279dc', 'sg_uuid_list': ['2b857d45-cecf-440b-a7cf-436d4e16a1fb'], 'static_route_list': None, 'vm_project_uuid': '2bc25706-3e1b-4dcd-bbbb-0261d020b882', 'admin_state': 'Enabled', 'flow_key_idx': '27', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-11 04:14:17,601 - DEBUG - Starting Layer 2 verification in Agent 2025-12-11 04:14:17,611 - DEBUG - Layer 2 path is seen for VM MAC 02:3e:8c:2d:36:9e in agent 10.0.0.23 2025-12-11 04:14:17,611 - DEBUG - Active layer 2 route in agent is present for VMI tap3e8c2d36-9e 2025-12-11 04:14:17,611 - DEBUG - L2 label(29) matches bw route table and interface table 2025-12-11 04:14:17,616 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is found in Agent of node 10.0.0.141 2025-12-11 04:14:17,738 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is consistent in agent 10.0.0.141 2025-12-11 04:14:17,738 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.141 2025-12-11 04:14:17,738 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 verification for VM ctest-vn1_vm1-57438079 in Agent 10.0.0.141 passed 2025-12-11 04:14:17,846 - DEBUG - Starting all layer 2 verification in agent 10.0.0.141 2025-12-11 04:14:17,857 - DEBUG - Route for VM MAC 02:3e:8c:2d:36:9e is consistent in agent 10.0.0.141 2025-12-11 04:14:17,864 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is found in Agent of node 10.0.0.23 2025-12-11 04:14:17,985 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is consistent in agent 10.0.0.23 2025-12-11 04:14:17,985 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.23 2025-12-11 04:14:17,985 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 verification for VM ctest-vn1_vm1-57438079 in Agent 10.0.0.23 passed 2025-12-11 04:14:18,093 - DEBUG - Starting all layer 2 verification in agent 10.0.0.23 2025-12-11 04:14:18,105 - DEBUG - Route for VM MAC 02:3e:8c:2d:36:9e is consistent in agent 10.0.0.23 2025-12-11 04:14:18,105 - INFO - VM ctest-vn1_vm1-57438079 verifications in Compute nodes passed 2025-12-11 04:14:18,319 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.141 2025-12-11 04:14:18,324 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.241, 24; Got 10.20.0.241,24 2025-12-11 04:14:18,324 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.141 2025-12-11 04:14:18,433 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.23 2025-12-11 04:14:18,440 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-12-11 04:14:18,441 - INFO - Validated routes of VM ctest-vn1_vm1-57438079 in all vrouters 2025-12-11 04:14:18,929 - DEBUG - Starting all layer2 verification in 10.0.0.61 Control Node 2025-12-11 04:14:19,049 - DEBUG - Layer2 route found for VM MAC 02:3e:8c:2d:36:9e in Control-node 10.0.0.61 2025-12-11 04:14:19,049 - DEBUG - L2 Label for VM ctest-vn1_vm1-57438079 same between Control-node 10.0.0.61 and Agent, Expected: 29, Seen: 29 2025-12-11 04:14:19,049 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-12-11 04:14:19,188 - DEBUG - Layer2 route found for VM MAC 02:3e:8c:2d:36:9e in Control-node 10.0.0.27 2025-12-11 04:14:19,188 - DEBUG - L2 Label for VM ctest-vn1_vm1-57438079 same between Control-node 10.0.0.27 and Agent, Expected: 29, Seen: 29 2025-12-11 04:14:19,188 - INFO - Verification in Control-nodes for VM ctest-vn1_vm1-57438079 passed 2025-12-11 04:14:19,188 - DEBUG - Verifying the vm in opserver 2025-12-11 04:14:19,188 - DEBUG - Verifying in collector 10.0.0.27 ... 2025-12-11 04:14:19,188 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77?flat 2025-12-11 04:14:19,247 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-89570916:3e8c2d36-9ecd-4f00-96c7-1066028279dc?flat 2025-12-11 04:14:19,307 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-89570916:3e8c2d36-9ecd-4f00-96c7-1066028279dc?flat 2025-12-11 04:14:19,326 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77?flat 2025-12-11 04:14:19,343 - DEBUG - Verifying vm in vn uve 2025-12-11 04:14:19,343 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-89570916:3e8c2d36-9ecd-4f00-96c7-1066028279dc?flat 2025-12-11 04:14:19,408 - DEBUG - VM uve shows interface as {'rx_vlan': 65535, 'active': True, 'ip6_address': '::', 'admin_state': True, 'l2_active': True, 'mac_address': '02:3e:8c:2d:36:9e', '__T': 1765426455000272, 'vm_name': 'ctest-vn1_vm1-57438079', 'tx_vlan': 65535, 'ip4_active': True, 'gateway': '192.168.1.1', 'ip6_active': False, 'vn_uuid': '3cefc596-4b06-4e66-8c15-44705d46adb5', 'fixed_ip4_list': ['192.168.1.3'], 'label': 24, 'is_health_check_active': True, 'vm_uuid': '034da2e8-ef2b-4480-a910-a102aa142f77', 'port_mirror_enabled': False, 'ip_address': '192.168.1.3', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405', 'uuid': '3e8c2d36-9ecd-4f00-96c7-1066028279dc', 'vhostuser_mode': 0} 2025-12-11 04:14:19,408 - DEBUG - VM uve shows ip address as ['192.168.1.3', '::'] 2025-12-11 04:14:19,408 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 2025-12-11 04:14:19,408 - INFO - Verifying through opserver in 10.0.0.61 2025-12-11 04:14:19,408 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405?flat 2025-12-11 04:14:19,439 - DEBUG - expected vm list 034da2e8-ef2b-4480-a910-a102aa142f77 2025-12-11 04:14:19,439 - DEBUG - Extracted vm list ['034da2e8-ef2b-4480-a910-a102aa142f77'] 2025-12-11 04:14:19,439 - INFO - VM 034da2e8-ef2b-4480-a910-a102aa142f77 is present in default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 2025-12-11 04:14:19,439 - DEBUG - Verifying vm in vrouter uve 2025-12-11 04:14:19,439 - DEBUG - Getting info from collector 10.0.0.27.. 2025-12-11 04:14:19,439 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77?flat 2025-12-11 04:14:19,501 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4883-2'] 2025-12-11 04:14:19,502 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-2:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:14:19,560 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-2?flat 2025-12-11 04:14:19,592 - DEBUG - VM 034da2e8-ef2b-4480-a910-a102aa142f77 is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-2 uve 2025-12-11 04:14:19,592 - DEBUG - Validated that VM ctest-vn1_vm1-57438079 is in Vrouter an-jenkins-deploy-platform-ansible-os-4883-2 UVE 2025-12-11 04:14:19,592 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-12-11 04:14:19,600 - DEBUG - Expected tap interface of VM uuid 034da2e8-ef2b-4480-a910-a102aa142f77 is default-domain:ctest-TestBasicVMVN0-89570916:3e8c2d36-9ecd-4f00-96c7-1066028279dc 2025-12-11 04:14:19,600 - DEBUG - Expected VN of VM uuid 034da2e8-ef2b-4480-a910-a102aa142f77 is default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 2025-12-11 04:14:19,600 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-2:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:14:19,646 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-2?flat 2025-12-11 04:14:19,676 - DEBUG - VM 034da2e8-ef2b-4480-a910-a102aa142f77 is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-2 uve 2025-12-11 04:14:19,676 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-89570916:3e8c2d36-9ecd-4f00-96c7-1066028279dc of vm 034da2e8-ef2b-4480-a910-a102aa142f77 is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-2 uve 2025-12-11 04:14:19,676 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 of vm 034da2e8-ef2b-4480-a910-a102aa142f77 is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-2 uve 2025-12-11 04:14:19,676 - DEBUG - Verifying the 034da2e8-ef2b-4480-a910-a102aa142f77 virtual network link through opserver 10.0.0.61 2025-12-11 04:14:19,676 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machines 2025-12-11 04:14:19,689 - DEBUG - VM link and name as {'name': '034da2e8-ef2b-4480-a910-a102aa142f77', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77?flat'} 2025-12-11 04:14:19,690 - DEBUG - Verifying the 034da2e8-ef2b-4480-a910-a102aa142f77 virtual network link through opserver 10.0.0.33 2025-12-11 04:14:19,690 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-11 04:14:19,703 - DEBUG - VM link and name as {'name': '034da2e8-ef2b-4480-a910-a102aa142f77', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77?flat'} 2025-12-11 04:14:19,703 - DEBUG - Verifying the 034da2e8-ef2b-4480-a910-a102aa142f77 virtual network link through opserver 10.0.0.27 2025-12-11 04:14:19,703 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-11 04:14:19,715 - DEBUG - VM link and name as {'name': '034da2e8-ef2b-4480-a910-a102aa142f77', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77?flat'} 2025-12-11 04:14:19,715 - INFO - VM ctest-vn1_vm1-57438079 validations in Opserver passed 2025-12-11 04:14:19,794 - DEBUG - VM is in ACTIVE state now 2025-12-11 04:14:19,794 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:14:19,794 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 2025-12-11 04:14:19,802 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/28a11ee5-1428-45cc-9116-03dd0e8f7a63 2025-12-11 04:14:19,815 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 2025-12-11 04:14:19,825 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/28a11ee5-1428-45cc-9116-03dd0e8f7a63 2025-12-11 04:14:19,837 - DEBUG - Requesting: http://10.0.0.27:8082/instance-ip/95392436-fd86-4ed7-80f5-1826626c1187 2025-12-11 04:14:19,847 - DEBUG - Verifying in api server 10.0.0.27 2025-12-11 04:14:19,847 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 2025-12-11 04:14:19,854 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/28a11ee5-1428-45cc-9116-03dd0e8f7a63 2025-12-11 04:14:19,865 - DEBUG - Requesting: http://10.0.0.27:8082/instance-ip/95392436-fd86-4ed7-80f5-1826626c1187 2025-12-11 04:14:19,877 - INFO - VM ctest-vn1_vm2-51220443 verfication in all API Servers passed 2025-12-11 04:14:19,902 - DEBUG - VM ctest-vn1_vm2-51220443 Tap interface: {'index': '3', 'name': 'tap28a11ee5-14', 'uuid': '28a11ee5-1428-45cc-9116-03dd0e8f7a63', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405:ctest-vn1-31264405', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405', 'vm_uuid': '81f16b9d-c76b-4522-9f5c-b2e71d6a4e79', 'vm_name': 'ctest-vn1_vm2-51220443', 'ip_addr': '192.168.1.4', 'mac_addr': '02:28:a1:1e:e5:14', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-89570916:28a11ee5-1428-45cc-9116-03dd0e8f7a63', 'sg_uuid_list': ['2b857d45-cecf-440b-a7cf-436d4e16a1fb'], 'static_route_list': None, 'vm_project_uuid': '2bc25706-3e1b-4dcd-bbbb-0261d020b882', 'admin_state': 'Enabled', 'flow_key_idx': '23', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-11 04:14:19,902 - DEBUG - Agent 10.0.0.141 vrf name: default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405:ctest-vn1-31264405 2025-12-11 04:14:19,908 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405:ctest-vn1-31264405', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405', 'table_label': '-1', 'vxlan_id': '14', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.249:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-12-11 04:14:19,993 - DEBUG - VM ctest-vn1_vm2-51220443 : Tap interface tap28a11ee5-14 is set to Active 2025-12-11 04:14:19,993 - DEBUG - Tap interface tap28a11ee5-14 detail : {'index': '3', 'name': 'tap28a11ee5-14', 'uuid': '28a11ee5-1428-45cc-9116-03dd0e8f7a63', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405:ctest-vn1-31264405', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405', 'vm_uuid': '81f16b9d-c76b-4522-9f5c-b2e71d6a4e79', 'vm_name': 'ctest-vn1_vm2-51220443', 'ip_addr': '192.168.1.4', 'mac_addr': '02:28:a1:1e:e5:14', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-89570916:28a11ee5-1428-45cc-9116-03dd0e8f7a63', 'sg_uuid_list': ['2b857d45-cecf-440b-a7cf-436d4e16a1fb'], 'static_route_list': None, 'vm_project_uuid': '2bc25706-3e1b-4dcd-bbbb-0261d020b882', 'admin_state': 'Enabled', 'flow_key_idx': '23', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-11 04:14:20,108 - DEBUG - Starting Layer 2 verification in Agent 2025-12-11 04:14:20,120 - DEBUG - Layer 2 path is seen for VM MAC 02:28:a1:1e:e5:14 in agent 10.0.0.141 2025-12-11 04:14:20,121 - DEBUG - Active layer 2 route in agent is present for VMI tap28a11ee5-14 2025-12-11 04:14:20,121 - DEBUG - L2 label(29) matches bw route table and interface table 2025-12-11 04:14:20,130 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is found in Agent of node 10.0.0.141 2025-12-11 04:14:20,257 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is consistent in agent 10.0.0.141 2025-12-11 04:14:20,257 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.141 2025-12-11 04:14:20,257 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 verification for VM ctest-vn1_vm2-51220443 in Agent 10.0.0.141 passed 2025-12-11 04:14:20,365 - DEBUG - Starting all layer 2 verification in agent 10.0.0.141 2025-12-11 04:14:20,375 - DEBUG - Route for VM MAC 02:28:a1:1e:e5:14 is consistent in agent 10.0.0.141 2025-12-11 04:14:20,381 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is found in Agent of node 10.0.0.23 2025-12-11 04:14:20,507 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 is consistent in agent 10.0.0.23 2025-12-11 04:14:20,507 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.23 2025-12-11 04:14:20,507 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 verification for VM ctest-vn1_vm2-51220443 in Agent 10.0.0.23 passed 2025-12-11 04:14:20,616 - DEBUG - Starting all layer 2 verification in agent 10.0.0.23 2025-12-11 04:14:20,628 - DEBUG - Route for VM MAC 02:28:a1:1e:e5:14 is consistent in agent 10.0.0.23 2025-12-11 04:14:20,628 - INFO - VM ctest-vn1_vm2-51220443 verifications in Compute nodes passed 2025-12-11 04:14:20,854 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.141 2025-12-11 04:14:20,859 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-12-11 04:14:20,969 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.23 2025-12-11 04:14:20,976 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.249, 24; Got 10.20.0.249,24 2025-12-11 04:14:20,976 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.23 2025-12-11 04:14:20,976 - INFO - Validated routes of VM ctest-vn1_vm2-51220443 in all vrouters 2025-12-11 04:14:21,493 - DEBUG - Starting all layer2 verification in 10.0.0.61 Control Node 2025-12-11 04:14:21,621 - DEBUG - Layer2 route found for VM MAC 02:28:a1:1e:e5:14 in Control-node 10.0.0.61 2025-12-11 04:14:21,621 - DEBUG - L2 Label for VM ctest-vn1_vm2-51220443 same between Control-node 10.0.0.61 and Agent, Expected: 29, Seen: 29 2025-12-11 04:14:21,621 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-12-11 04:14:21,749 - DEBUG - Layer2 route found for VM MAC 02:28:a1:1e:e5:14 in Control-node 10.0.0.27 2025-12-11 04:14:21,749 - DEBUG - L2 Label for VM ctest-vn1_vm2-51220443 same between Control-node 10.0.0.27 and Agent, Expected: 29, Seen: 29 2025-12-11 04:14:21,749 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-51220443 passed 2025-12-11 04:14:21,749 - DEBUG - Verifying the vm in opserver 2025-12-11 04:14:21,749 - DEBUG - Verifying in collector 10.0.0.27 ... 2025-12-11 04:14:21,749 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine/81f16b9d-c76b-4522-9f5c-b2e71d6a4e79?flat 2025-12-11 04:14:21,766 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-89570916:28a11ee5-1428-45cc-9116-03dd0e8f7a63?flat 2025-12-11 04:14:21,788 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-89570916:28a11ee5-1428-45cc-9116-03dd0e8f7a63?flat 2025-12-11 04:14:21,807 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine/81f16b9d-c76b-4522-9f5c-b2e71d6a4e79?flat 2025-12-11 04:14:21,823 - DEBUG - Verifying vm in vn uve 2025-12-11 04:14:21,823 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-89570916:28a11ee5-1428-45cc-9116-03dd0e8f7a63?flat 2025-12-11 04:14:21,842 - DEBUG - VM uve shows interface as {'port_mirror_enabled': False, '__T': 1765426446504362, 'fixed_ip4_list': ['192.168.1.4'], 'admin_state': True, 'label': 24, 'ip6_active': False, 'rx_vlan': 65535, 'gateway': '192.168.1.1', 'vn_uuid': '3cefc596-4b06-4e66-8c15-44705d46adb5', 'ip4_active': True, 'mac_address': '02:28:a1:1e:e5:14', 'is_health_check_active': True, 'ip6_address': '::', 'ip_address': '192.168.1.4', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405', 'tx_vlan': 65535, 'l2_active': True, 'uuid': '28a11ee5-1428-45cc-9116-03dd0e8f7a63', 'active': True, 'vm_name': 'ctest-vn1_vm2-51220443', 'vhostuser_mode': 0, 'vm_uuid': '81f16b9d-c76b-4522-9f5c-b2e71d6a4e79'} 2025-12-11 04:14:21,842 - DEBUG - VM uve shows ip address as ['192.168.1.4', '::'] 2025-12-11 04:14:21,842 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 2025-12-11 04:14:21,842 - INFO - Verifying through opserver in 10.0.0.61 2025-12-11 04:14:21,842 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405?flat 2025-12-11 04:14:21,872 - DEBUG - expected vm list 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 2025-12-11 04:14:21,872 - DEBUG - Extracted vm list ['81f16b9d-c76b-4522-9f5c-b2e71d6a4e79'] 2025-12-11 04:14:21,872 - INFO - VM 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 is present in default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 2025-12-11 04:14:21,872 - DEBUG - Verifying vm in vrouter uve 2025-12-11 04:14:21,872 - DEBUG - Getting info from collector 10.0.0.27.. 2025-12-11 04:14:21,872 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine/81f16b9d-c76b-4522-9f5c-b2e71d6a4e79?flat 2025-12-11 04:14:21,888 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4883-1'] 2025-12-11 04:14:21,888 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-1:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:14:21,932 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-1?flat 2025-12-11 04:14:22,014 - DEBUG - VM 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:14:22,015 - DEBUG - Validated that VM ctest-vn1_vm2-51220443 is in Vrouter an-jenkins-deploy-platform-ansible-os-4883-1 UVE 2025-12-11 04:14:22,015 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-12-11 04:14:22,022 - DEBUG - Expected tap interface of VM uuid 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 is default-domain:ctest-TestBasicVMVN0-89570916:28a11ee5-1428-45cc-9116-03dd0e8f7a63 2025-12-11 04:14:22,022 - DEBUG - Expected VN of VM uuid 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 is default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 2025-12-11 04:14:22,022 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-1:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:14:22,063 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-1?flat 2025-12-11 04:14:22,144 - DEBUG - VM 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:14:22,144 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-89570916:28a11ee5-1428-45cc-9116-03dd0e8f7a63 of vm 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:14:22,144 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 of vm 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:14:22,144 - DEBUG - Verifying the 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 virtual network link through opserver 10.0.0.61 2025-12-11 04:14:22,144 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machines 2025-12-11 04:14:22,156 - DEBUG - VM link and name as {'name': '81f16b9d-c76b-4522-9f5c-b2e71d6a4e79', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-machine/81f16b9d-c76b-4522-9f5c-b2e71d6a4e79?flat'} 2025-12-11 04:14:22,156 - DEBUG - Verifying the 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 virtual network link through opserver 10.0.0.33 2025-12-11 04:14:22,156 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-11 04:14:22,167 - DEBUG - VM link and name as {'name': '81f16b9d-c76b-4522-9f5c-b2e71d6a4e79', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/81f16b9d-c76b-4522-9f5c-b2e71d6a4e79?flat'} 2025-12-11 04:14:22,168 - DEBUG - Verifying the 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 virtual network link through opserver 10.0.0.27 2025-12-11 04:14:22,168 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-11 04:14:22,179 - DEBUG - VM link and name as {'name': '81f16b9d-c76b-4522-9f5c-b2e71d6a4e79', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/81f16b9d-c76b-4522-9f5c-b2e71d6a4e79?flat'} 2025-12-11 04:14:22,179 - INFO - VM ctest-vn1_vm2-51220443 validations in Opserver passed 2025-12-11 04:14:22,179 - INFO - Checking the ping between the VM again 2025-12-11 04:14:22,180 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:14:22,180 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.23, gateway password: c0ntrail123 2025-12-11 04:14:22,180 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-12-11 04:14:24,664 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=5.32 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.26 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.22 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2005ms rtt min/avg/max/mdev = 1.225/2.607/5.328/1.924 ms 2025-12-11 04:14:24,664 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-57438079 passed 2025-12-11 04:14:24,664 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:14:24,664 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.141, gateway password: c0ntrail123 2025-12-11 04:14:24,664 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-12-11 04:14:27,200 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=3.18 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.06 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.52 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.069/1.924/3.184/0.910 ms 2025-12-11 04:14:27,200 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-51220443 passed 2025-12-11 04:14:27,200 - INFO - Deleting VM ctest-vn1_vm2-51220443 2025-12-11 04:14:27,265 - DEBUG - Verifying in api server 10.0.0.27 2025-12-11 04:14:27,265 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 2025-12-11 04:14:27,275 - DEBUG - VM ID 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 of VM ctest-vn1_vm2-51220443 is still found in API Server 2025-12-11 04:14:29,275 - DEBUG - Verifying in api server 10.0.0.27 2025-12-11 04:14:29,276 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 2025-12-11 04:14:29,285 - DEBUG - Response Code: 404 2025-12-11 04:14:29,285 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 2025-12-11 04:14:29,290 - DEBUG - Response Code: 404 2025-12-11 04:14:29,290 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 2025-12-11 04:14:29,295 - DEBUG - Response Code: 404 2025-12-11 04:14:29,295 - INFO - VM ctest-vn1_vm2-51220443 is fully removed in API-Server 2025-12-11 04:14:29,448 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-12-11 04:14:29,483 - DEBUG - Validated that vrouter 10.0.0.141 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 2025-12-11 04:14:29,591 - DEBUG - VM route 192.168.1.4 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 of compute 10.0.0.23 2025-12-11 04:14:29,596 - DEBUG - Validated that vrouter 10.0.0.23 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 2025-12-11 04:14:29,596 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 2025-12-11 04:14:29,596 - INFO - VM ctest-vn1_vm2-51220443 is removed in Compute, and routes are removed in all compute nodes 2025-12-11 04:14:29,842 - INFO - Routes for VM ctest-vn1_vm2-51220443 is removed in all control-nodes 2025-12-11 04:14:29,842 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-1:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:14:29,885 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-1?flat 2025-12-11 04:14:29,920 - DEBUG - VM 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:14:29,920 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-11 04:14:29,920 - DEBUG - interface for vm 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 deleted from vrouter uve ... 2025-12-11 04:14:29,920 - DEBUG - Deleted interface not in error interface list ... 2025-12-11 04:14:32,921 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-1:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:14:32,965 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-1?flat 2025-12-11 04:14:32,996 - DEBUG - VM 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:14:32,996 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-11 04:14:32,996 - DEBUG - interface for vm 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 deleted from vrouter uve ... 2025-12-11 04:14:32,996 - DEBUG - Deleted interface not in error interface list ... 2025-12-11 04:14:35,997 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-1:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:14:36,042 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-1?flat 2025-12-11 04:14:36,072 - DEBUG - VM 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:14:36,072 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-11 04:14:36,072 - DEBUG - interface for vm 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 deleted from vrouter uve ... 2025-12-11 04:14:36,072 - DEBUG - Deleted interface not in error interface list ... 2025-12-11 04:14:39,073 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-1:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:14:39,117 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-1?flat 2025-12-11 04:14:39,152 - DEBUG - VM 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:14:39,152 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-11 04:14:39,152 - DEBUG - interface for vm 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 deleted from vrouter uve ... 2025-12-11 04:14:39,153 - DEBUG - Deleted interface not in error interface list ... 2025-12-11 04:14:39,153 - DEBUG - Validated that VM 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 is removed in Vrouter UVE 2025-12-11 04:14:39,153 - DEBUG - Verifying through opserver in 10.0.0.61 2025-12-11 04:14:39,153 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405?flat 2025-12-11 04:14:39,180 - DEBUG - 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 BM not in default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 uve 2025-12-11 04:14:39,180 - DEBUG - Validated that VM 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 is not present in the VN UVE 2025-12-11 04:14:39,180 - DEBUG - Verifying the 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 virtual network link through opserver 10.0.0.61 2025-12-11 04:14:39,180 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machines 2025-12-11 04:14:39,192 - INFO - 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 vm uve deleted from opserver 2025-12-11 04:14:39,192 - DEBUG - Verifying the 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 virtual network link through opserver 10.0.0.33 2025-12-11 04:14:39,192 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-11 04:14:39,207 - INFO - 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 vm uve deleted from opserver 2025-12-11 04:14:39,207 - DEBUG - Verifying the 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 virtual network link through opserver 10.0.0.27 2025-12-11 04:14:39,207 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-11 04:14:39,238 - INFO - 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 vm uve deleted from opserver 2025-12-11 04:14:39,238 - INFO - Validated that VM 81f16b9d-c76b-4522-9f5c-b2e71d6a4e79 is removed from Opserver 2025-12-11 04:14:39,238 - INFO - Deleting VM ctest-vn1_vm1-57438079 2025-12-11 04:14:39,321 - DEBUG - Verifying in api server 10.0.0.27 2025-12-11 04:14:39,322 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77 2025-12-11 04:14:39,330 - DEBUG - VM ID 034da2e8-ef2b-4480-a910-a102aa142f77 of VM ctest-vn1_vm1-57438079 is still found in API Server 2025-12-11 04:14:41,330 - DEBUG - Verifying in api server 10.0.0.27 2025-12-11 04:14:41,331 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77 2025-12-11 04:14:41,337 - DEBUG - Response Code: 404 2025-12-11 04:14:41,337 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77 2025-12-11 04:14:41,342 - DEBUG - Response Code: 404 2025-12-11 04:14:41,343 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/034da2e8-ef2b-4480-a910-a102aa142f77 2025-12-11 04:14:41,348 - DEBUG - Response Code: 404 2025-12-11 04:14:41,348 - INFO - VM ctest-vn1_vm1-57438079 is fully removed in API-Server 2025-12-11 04:14:41,500 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-12-11 04:14:41,538 - DEBUG - Validated that vrouter 10.0.0.141 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 2025-12-11 04:14:41,547 - DEBUG - Validated that vrouter 10.0.0.23 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 2025-12-11 04:14:41,548 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 2025-12-11 04:14:41,548 - INFO - VM ctest-vn1_vm1-57438079 is removed in Compute, and routes are removed in all compute nodes 2025-12-11 04:14:41,784 - INFO - Routes for VM ctest-vn1_vm1-57438079 is removed in all control-nodes 2025-12-11 04:14:41,784 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-2:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:14:41,833 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-2?flat 2025-12-11 04:14:41,876 - DEBUG - VM 034da2e8-ef2b-4480-a910-a102aa142f77 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4883-2 uve 2025-12-11 04:14:41,876 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-11 04:14:41,876 - DEBUG - interface for vm 034da2e8-ef2b-4480-a910-a102aa142f77 deleted from vrouter uve ... 2025-12-11 04:14:41,876 - DEBUG - Deleted interface not in error interface list ... 2025-12-11 04:14:44,878 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-2:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:14:44,923 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-2?flat 2025-12-11 04:14:44,952 - DEBUG - VM 034da2e8-ef2b-4480-a910-a102aa142f77 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4883-2 uve 2025-12-11 04:14:44,952 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-11 04:14:44,952 - DEBUG - interface for vm 034da2e8-ef2b-4480-a910-a102aa142f77 deleted from vrouter uve ... 2025-12-11 04:14:44,952 - DEBUG - Deleted interface not in error interface list ... 2025-12-11 04:14:47,953 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-2:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:14:47,999 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-2?flat 2025-12-11 04:14:48,030 - DEBUG - VM 034da2e8-ef2b-4480-a910-a102aa142f77 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4883-2 uve 2025-12-11 04:14:48,030 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-11 04:14:48,030 - DEBUG - interface for vm 034da2e8-ef2b-4480-a910-a102aa142f77 deleted from vrouter uve ... 2025-12-11 04:14:48,030 - DEBUG - Deleted interface not in error interface list ... 2025-12-11 04:14:48,030 - DEBUG - Validated that VM 034da2e8-ef2b-4480-a910-a102aa142f77 is removed in Vrouter UVE 2025-12-11 04:14:48,030 - DEBUG - Verifying through opserver in 10.0.0.61 2025-12-11 04:14:48,031 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405?flat 2025-12-11 04:14:48,047 - DEBUG - 034da2e8-ef2b-4480-a910-a102aa142f77 BM not in default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-31264405 uve 2025-12-11 04:14:48,047 - DEBUG - Validated that VM 034da2e8-ef2b-4480-a910-a102aa142f77 is not present in the VN UVE 2025-12-11 04:14:48,047 - DEBUG - Verifying the 034da2e8-ef2b-4480-a910-a102aa142f77 virtual network link through opserver 10.0.0.61 2025-12-11 04:14:48,047 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machines 2025-12-11 04:14:48,060 - INFO - 034da2e8-ef2b-4480-a910-a102aa142f77 vm uve deleted from opserver 2025-12-11 04:14:48,060 - DEBUG - Verifying the 034da2e8-ef2b-4480-a910-a102aa142f77 virtual network link through opserver 10.0.0.33 2025-12-11 04:14:48,060 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-11 04:14:48,072 - INFO - 034da2e8-ef2b-4480-a910-a102aa142f77 vm uve deleted from opserver 2025-12-11 04:14:48,072 - DEBUG - Verifying the 034da2e8-ef2b-4480-a910-a102aa142f77 virtual network link through opserver 10.0.0.27 2025-12-11 04:14:48,072 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-11 04:14:48,085 - INFO - 034da2e8-ef2b-4480-a910-a102aa142f77 vm uve deleted from opserver 2025-12-11 04:14:48,085 - INFO - Validated that VM 034da2e8-ef2b-4480-a910-a102aa142f77 is removed from Opserver 2025-12-11 04:14:48,085 - INFO - Deleting VN ctest-vn1-31264405 2025-12-11 04:14:48,267 - DEBUG - Response for deleting network () 2025-12-11 04:14:48,267 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/3664c295-184e-47d1-9c3d-bfdf79b26b1d 2025-12-11 04:14:48,272 - DEBUG - Response Code: 404 2025-12-11 04:14:48,272 - DEBUG - Requesting: http://10.0.0.61:8082/domains 2025-12-11 04:14:48,279 - DEBUG - Requesting: http://10.0.0.61:8082/domain/35136ed0-c5bb-4823-8f5c-ae1b4c17652e 2025-12-11 04:14:48,337 - DEBUG - Requesting: http://10.0.0.61:8082/project/2bc25706-3e1b-4dcd-bbbb-0261d020b882 2025-12-11 04:14:48,415 - INFO - Validated that VN ctest-vn1-31264405 is not found in API Server 2025-12-11 04:14:48,426 - DEBUG - VN ctest-vn1-31264405 is not present in Agent 10.0.0.141 2025-12-11 04:14:48,436 - DEBUG - VN ctest-vn1-31264405 is not present in Agent 10.0.0.23 2025-12-11 04:14:48,436 - INFO - Validated that VN ctest-vn1-31264405 is not in any agent 2025-12-11 04:14:48,464 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-31264405 info 2025-12-11 04:14:49,715 - DEBUG - Skipping xmpp flap check 2025-12-11 04:14:49,715 - INFO - END TEST : test_control_node_switchover : PASSED[0:03:01] 2025-12-11 04:14:49,715 - INFO - -------------------------------------------------------------------------------- 2025-12-11 04:14:49,719 - INFO - ================================================================================ 2025-12-11 04:14:49,719 - INFO - STARTING TEST : test_process_restart_in_policy_between_vns 2025-12-11 04:14:49,719 - INFO - TEST DESCRIPTION : Test to validate that with policy having rule to check icmp fwding between VMs on different VNs , ping between VMs should pass with process restarts 1. Pick 2 VN's from resource pool which has one VM each 2. Create policy with icmp allow rule between those VN's and bind it networks 3. Ping from one VM to another VM 4. Restart process 'vrouter' and 'control' on setup 5. Ping again between VM's after process restart Pass criteria: Step 2,3,4 and 5 should pass 2025-12-11 04:14:50,988 - DEBUG - Skipping xmpp flap check 2025-12-11 04:14:50,988 - INFO - Initial checks done. Running the testcase now 2025-12-11 04:14:50,988 - INFO - 2025-12-11 04:14:50,995 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-12-11 04:14:51,284 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-12-11 04:14:51,804 - INFO - Created VN ctest-vn1-77140010, UUID :bc21472c-b067-4b88-9e30-60838e877e53 2025-12-11 04:14:52,109 - DEBUG - Requesting: http://10.0.0.61:8082/domains 2025-12-11 04:14:52,114 - DEBUG - Requesting: http://10.0.0.61:8082/domain/35136ed0-c5bb-4823-8f5c-ae1b4c17652e 2025-12-11 04:14:52,124 - DEBUG - Requesting: http://10.0.0.61:8082/project/2bc25706-3e1b-4dcd-bbbb-0261d020b882 2025-12-11 04:14:52,226 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/bc21472c-b067-4b88-9e30-60838e877e53 2025-12-11 04:14:52,238 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/bc21472c-b067-4b88-9e30-60838e877e53 2025-12-11 04:14:52,249 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/eda0b80c-f51b-4438-a2ae-ffae70fd182f 2025-12-11 04:14:52,261 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/eda0b80c-f51b-4438-a2ae-ffae70fd182f 2025-12-11 04:14:52,270 - DEBUG - Requesting: http://10.0.0.61:8082/route-target/2c6bea52-5e41-4cf9-b11c-31070197cffb 2025-12-11 04:14:52,277 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-11 04:14:52,278 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/bc21472c-b067-4b88-9e30-60838e877e53 2025-12-11 04:14:52,289 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/eda0b80c-f51b-4438-a2ae-ffae70fd182f 2025-12-11 04:14:52,307 - INFO - Verified VN network id 14 for VN bc21472c-b067-4b88-9e30-60838e877e53 2025-12-11 04:14:52,307 - INFO - Verifications in API Server for VN ctest-vn1-77140010 passed 2025-12-11 04:14:52,307 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/bc21472c-b067-4b88-9e30-60838e877e53 2025-12-11 04:14:52,320 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/eda0b80c-f51b-4438-a2ae-ffae70fd182f 2025-12-11 04:14:52,330 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/eda0b80c-f51b-4438-a2ae-ffae70fd182f 2025-12-11 04:14:52,340 - DEBUG - Requesting: http://10.0.0.61:8082/route-target/2c6bea52-5e41-4cf9-b11c-31070197cffb 2025-12-11 04:14:52,353 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-89570916', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13556194610625596296', 'uuid-lslong': '11398716775022952019'}, 'enable': 'true', 'created': '2025-12-11T04:14:51', 'last-modified': '2025-12-11T04:14:51', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.416350'} 2025-12-11 04:14:52,358 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-11 04:14:52,363 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-89570916', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13556194610625596296', 'uuid-lslong': '11398716775022952019'}, 'enable': 'true', 'created': '2025-12-11T04:14:51', 'last-modified': '2025-12-11T04:14:51', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.426750'} 2025-12-11 04:14:52,369 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-11 04:14:52,373 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-89570916', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13556194610625596296', 'uuid-lslong': '11398716775022952019'}, 'enable': 'true', 'created': '2025-12-11T04:14:51', 'last-modified': '2025-12-11T04:14:51', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.434565'} 2025-12-11 04:14:52,378 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-11 04:14:52,378 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-77140010 passed 2025-12-11 04:14:52,378 - DEBUG - ====Verifying policy data for ctest-vn1-77140010 in API_Server ====== 2025-12-11 04:14:52,379 - DEBUG - Requesting: http://10.0.0.61:8082/domains 2025-12-11 04:14:52,385 - DEBUG - Requesting: http://10.0.0.61:8082/domain/35136ed0-c5bb-4823-8f5c-ae1b4c17652e 2025-12-11 04:14:52,397 - DEBUG - Requesting: http://10.0.0.61:8082/project/2bc25706-3e1b-4dcd-bbbb-0261d020b882 2025-12-11 04:14:52,490 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/bc21472c-b067-4b88-9e30-60838e877e53 2025-12-11 04:14:52,500 - DEBUG - =>VN ctest-vn1-77140010 has no policy to be verified 2025-12-11 04:14:52,500 - DEBUG - Verifying the vn in opserver 2025-12-11 04:14:52,500 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 virtual network link through opserver 10.0.0.61 2025-12-11 04:14:52,500 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-12-11 04:14:52,510 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010?flat'} 2025-12-11 04:14:52,510 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 virtual network link through opserver 10.0.0.33 2025-12-11 04:14:52,510 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-12-11 04:14:52,522 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010?flat'} 2025-12-11 04:14:52,522 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 virtual network link through opserver 10.0.0.27 2025-12-11 04:14:52,522 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-12-11 04:14:52,537 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010?flat'} 2025-12-11 04:14:52,537 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is found in opserver 2025-12-11 04:14:52,558 - DEBUG - Do not have enough data to verify VN in agent 2025-12-11 04:14:52,580 - DEBUG - VRF ids for VN ctest-vn1-77140010: {} 2025-12-11 04:14:53,228 - INFO - Created VN ctest-vn2-17817119, UUID :a62f0c86-883a-4ca5-b5bb-5e23d6af5ad1 2025-12-11 04:14:53,515 - DEBUG - Requesting: http://10.0.0.61:8082/domains 2025-12-11 04:14:53,521 - DEBUG - Requesting: http://10.0.0.61:8082/domain/35136ed0-c5bb-4823-8f5c-ae1b4c17652e 2025-12-11 04:14:53,531 - DEBUG - Requesting: http://10.0.0.61:8082/project/2bc25706-3e1b-4dcd-bbbb-0261d020b882 2025-12-11 04:14:53,622 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/a62f0c86-883a-4ca5-b5bb-5e23d6af5ad1 2025-12-11 04:14:53,635 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/a62f0c86-883a-4ca5-b5bb-5e23d6af5ad1 2025-12-11 04:14:53,644 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/bc35afab-d0bc-4963-98b0-ed3d87b5c12d 2025-12-11 04:14:53,654 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/bc35afab-d0bc-4963-98b0-ed3d87b5c12d 2025-12-11 04:14:53,660 - DEBUG - Requesting: http://10.0.0.61:8082/route-target/c881f928-1a47-465a-8c7d-5d089a001ecd 2025-12-11 04:14:53,666 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-12-11 04:14:53,666 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/a62f0c86-883a-4ca5-b5bb-5e23d6af5ad1 2025-12-11 04:14:53,674 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/bc35afab-d0bc-4963-98b0-ed3d87b5c12d 2025-12-11 04:14:53,687 - INFO - Verified VN network id 15 for VN a62f0c86-883a-4ca5-b5bb-5e23d6af5ad1 2025-12-11 04:14:53,687 - INFO - Verifications in API Server for VN ctest-vn2-17817119 passed 2025-12-11 04:14:53,688 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/a62f0c86-883a-4ca5-b5bb-5e23d6af5ad1 2025-12-11 04:14:53,696 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/bc35afab-d0bc-4963-98b0-ed3d87b5c12d 2025-12-11 04:14:53,703 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/bc35afab-d0bc-4963-98b0-ed3d87b5c12d 2025-12-11 04:14:53,710 - DEBUG - Requesting: http://10.0.0.61:8082/route-target/c881f928-1a47-465a-8c7d-5d089a001ecd 2025-12-11 04:14:53,722 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-89570916', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '11974803706152111269', 'uuid-lslong': '13095163849528531665'}, 'enable': 'true', 'created': '2025-12-11T04:14:53', 'last-modified': '2025-12-11T04:14:53', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.362371'} 2025-12-11 04:14:53,726 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-12-11 04:14:53,731 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-89570916', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '11974803706152111269', 'uuid-lslong': '13095163849528531665'}, 'enable': 'true', 'created': '2025-12-11T04:14:53', 'last-modified': '2025-12-11T04:14:53', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.368229'} 2025-12-11 04:14:53,738 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-12-11 04:14:53,744 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-89570916', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '11974803706152111269', 'uuid-lslong': '13095163849528531665'}, 'enable': 'true', 'created': '2025-12-11T04:14:53', 'last-modified': '2025-12-11T04:14:53', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.385638'} 2025-12-11 04:14:53,752 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-12-11 04:14:53,752 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-17817119 passed 2025-12-11 04:14:53,753 - DEBUG - ====Verifying policy data for ctest-vn2-17817119 in API_Server ====== 2025-12-11 04:14:53,753 - DEBUG - Requesting: http://10.0.0.61:8082/domains 2025-12-11 04:14:53,759 - DEBUG - Requesting: http://10.0.0.61:8082/domain/35136ed0-c5bb-4823-8f5c-ae1b4c17652e 2025-12-11 04:14:53,776 - DEBUG - Requesting: http://10.0.0.61:8082/project/2bc25706-3e1b-4dcd-bbbb-0261d020b882 2025-12-11 04:14:53,874 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/a62f0c86-883a-4ca5-b5bb-5e23d6af5ad1 2025-12-11 04:14:53,886 - DEBUG - =>VN ctest-vn2-17817119 has no policy to be verified 2025-12-11 04:14:53,886 - DEBUG - Verifying the vn in opserver 2025-12-11 04:14:53,886 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 virtual network link through opserver 10.0.0.61 2025-12-11 04:14:53,887 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-12-11 04:14:53,902 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119?flat'} 2025-12-11 04:14:53,902 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 virtual network link through opserver 10.0.0.33 2025-12-11 04:14:53,902 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-12-11 04:14:53,915 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119?flat'} 2025-12-11 04:14:53,915 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 virtual network link through opserver 10.0.0.27 2025-12-11 04:14:53,916 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-12-11 04:14:53,929 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119?flat'} 2025-12-11 04:14:53,929 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 is found in opserver 2025-12-11 04:14:53,956 - DEBUG - Do not have enough data to verify VN in agent 2025-12-11 04:14:53,965 - DEBUG - VRF ids for VN ctest-vn2-17817119: {} 2025-12-11 04:14:55,456 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4883-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4883-2) 2025-12-11 04:14:56,768 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4883-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4883-1) 2025-12-11 04:14:56,770 - INFO - Waiting for VM ctest-vn1_vm1-43073363 to be up.. 2025-12-11 04:14:56,848 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-12-11 04:15:01,936 - DEBUG - VM is in ACTIVE state now 2025-12-11 04:15:01,937 - INFO - VM name : ctest-vn1_vm1-43073363 2025-12-11 04:15:02,012 - DEBUG - VM ctest-vn1_vm1-43073363 ID is 14d2ab20-12c2-48a7-b1aa-5426c78d7c3c 2025-12-11 04:15:02,012 - DEBUG - VM ctest-vn1_vm1-43073363 launched on Node an-jenkins-deploy-platform-ansible-os-4883-2 2025-12-11 04:15:02,090 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/14d2ab20-12c2-48a7-b1aa-5426c78d7c3c 2025-12-11 04:15:02,100 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/5590a187-0957-4033-a3f5-19741fffe2d4 2025-12-11 04:15:05,304 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1034ms') 2025-12-11 04:15:05,305 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-43073363 failed! 2025-12-11 04:15:05,317 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:15:09,408 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms') 2025-12-11 04:15:09,408 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-43073363 failed! 2025-12-11 04:15:09,421 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:15:13,496 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-12-11 04:15:13,496 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-43073363 failed! 2025-12-11 04:15:13,509 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:15:17,593 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-12-11 04:15:17,593 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-43073363 failed! 2025-12-11 04:15:17,608 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:15:21,687 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-12-11 04:15:21,687 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-43073363 failed! 2025-12-11 04:15:21,700 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:15:25,794 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms') 2025-12-11 04:15:25,794 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-43073363 failed! 2025-12-11 04:15:25,807 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:15:29,879 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-12-11 04:15:29,880 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-43073363 failed! 2025-12-11 04:15:29,892 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:15:33,979 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-12-11 04:15:33,979 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-43073363 failed! 2025-12-11 04:15:33,994 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:15:38,075 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=7.78 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1016ms\r\nrtt min/avg/max/mdev = 7.782/7.782/7.782/0.000 ms') 2025-12-11 04:15:38,075 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-43073363 passed 2025-12-11 04:15:38,147 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:15:38,147 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-43073363, IP 192.168.1.252, Port 22 2025-12-11 04:15:38,213 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-11 04:15:38,295 - DEBUG - VM ctest-vn1_vm1-43073363 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-11 04:15:43,296 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:15:43,296 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-43073363, IP 192.168.1.252, Port 22 2025-12-11 04:15:43,351 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-11 04:15:43,433 - DEBUG - VM ctest-vn1_vm1-43073363 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-11 04:15:48,434 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:15:48,434 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-43073363, IP 192.168.1.252, Port 22 2025-12-11 04:15:48,501 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-11 04:15:48,583 - DEBUG - VM ctest-vn1_vm1-43073363 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-11 04:15:53,583 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:15:53,584 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-43073363, IP 192.168.1.252, Port 22 2025-12-11 04:15:53,639 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-11 04:15:53,712 - DEBUG - VM ctest-vn1_vm1-43073363 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-11 04:15:58,713 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:15:58,713 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-43073363, IP 192.168.1.252, Port 22 2025-12-11 04:15:58,881 - DEBUG - VM ctest-vn1_vm1-43073363 is ready for SSH connections 2025-12-11 04:15:58,881 - INFO - Waiting for VM ctest-vn2_vm1-34612927 to be up.. 2025-12-11 04:15:58,966 - DEBUG - VM is in ACTIVE state now 2025-12-11 04:15:58,966 - INFO - VM name : ctest-vn2_vm1-34612927 2025-12-11 04:15:59,048 - DEBUG - VM ctest-vn2_vm1-34612927 ID is 0db5c9d0-0106-4bbd-92d5-118de894105b 2025-12-11 04:15:59,048 - DEBUG - VM ctest-vn2_vm1-34612927 launched on Node an-jenkins-deploy-platform-ansible-os-4883-1 2025-12-11 04:15:59,126 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/0db5c9d0-0106-4bbd-92d5-118de894105b 2025-12-11 04:15:59,137 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/ca815bb4-a182-404c-9da1-e38ccc8bb731 2025-12-11 04:16:00,304 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=6.00 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=3.06 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 3.062/4.533/6.004/1.471 ms') 2025-12-11 04:16:00,304 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm1-34612927 passed 2025-12-11 04:16:00,372 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:16:00,373 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-34612927, IP 192.168.2.252, Port 22 2025-12-11 04:16:00,546 - DEBUG - VM ctest-vn2_vm1-34612927 is ready for SSH connections 2025-12-11 04:16:00,546 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:16:00,546 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.252, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.23, gateway password: c0ntrail123 2025-12-11 04:16:00,546 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.252 2025-12-11 04:16:21,752 - DEBUG - PING 192.168.2.252 (192.168.2.252) 56(84) bytes of data. 64 bytes from 192.168.2.252: icmp_req=1 ttl=63 time=8.62 ms 64 bytes from 192.168.2.252: icmp_req=2 ttl=63 time=2.04 ms 64 bytes from 192.168.2.252: icmp_req=3 ttl=63 time=1.08 ms --- 192.168.2.252 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.088/3.918/8.623/3.349 ms 2025-12-11 04:16:21,753 - INFO - Ping to IP 192.168.2.252 from VM ctest-vn1_vm1-43073363 passed 2025-12-11 04:16:25,204 - INFO - 10.0.0.141 2025-12-11 04:16:25,207 - INFO - agent:initializing 2025-12-11 04:16:25,207 - DEBUG - defaultdict(, {'10.0.0.141': {'agent': {'status': 'initializing', 'description': None}}}) 2025-12-11 04:16:25,208 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-11 04:16:30,308 - INFO - 10.0.0.141 2025-12-11 04:16:30,314 - INFO - agent:active 2025-12-11 04:16:30,314 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.141 2025-12-11 04:16:33,717 - INFO - 10.0.0.23 2025-12-11 04:16:33,720 - INFO - agent:initializing 2025-12-11 04:16:33,720 - DEBUG - defaultdict(, {'10.0.0.23': {'agent': {'status': 'initializing', 'description': None}}}) 2025-12-11 04:16:33,720 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-11 04:16:38,819 - INFO - 10.0.0.23 2025-12-11 04:16:38,825 - INFO - agent:active 2025-12-11 04:16:38,826 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.23 2025-12-11 04:16:39,446 - INFO - 10.0.0.61 2025-12-11 04:16:39,452 - INFO - control:initializing 2025-12-11 04:16:39,452 - DEBUG - defaultdict(, {'10.0.0.61': {'control': {'status': 'initializing', 'description': None}}}) 2025-12-11 04:16:39,453 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-11 04:16:44,549 - INFO - 10.0.0.61 2025-12-11 04:16:44,554 - INFO - control:active 2025-12-11 04:16:44,554 - INFO - Contrail services ['control'] are up on nodes 10.0.0.61 2025-12-11 04:16:44,966 - INFO - 10.0.0.33 2025-12-11 04:16:44,970 - INFO - control:initializing 2025-12-11 04:16:44,970 - DEBUG - defaultdict(, {'10.0.0.33': {'control': {'status': 'initializing', 'description': None}}}) 2025-12-11 04:16:44,970 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-11 04:16:50,062 - INFO - 10.0.0.33 2025-12-11 04:16:50,069 - INFO - control:active 2025-12-11 04:16:50,069 - INFO - Contrail services ['control'] are up on nodes 10.0.0.33 2025-12-11 04:16:50,472 - INFO - 10.0.0.27 2025-12-11 04:16:50,477 - INFO - control:initializing 2025-12-11 04:16:50,477 - DEBUG - defaultdict(, {'10.0.0.27': {'control': {'status': 'initializing', 'description': None}}}) 2025-12-11 04:16:50,477 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-11 04:16:55,578 - INFO - 10.0.0.27 2025-12-11 04:16:55,584 - INFO - control:active 2025-12-11 04:16:55,584 - INFO - Contrail services ['control'] are up on nodes 10.0.0.27 2025-12-11 04:16:56,358 - INFO - 10.0.0.61 2025-12-11 04:16:56,362 - INFO - api-server:initializing 2025-12-11 04:16:56,362 - DEBUG - defaultdict(, {'10.0.0.61': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-12-11 04:16:56,362 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-11 04:17:01,473 - INFO - 10.0.0.61 2025-12-11 04:17:01,477 - INFO - api-server:initializing 2025-12-11 04:17:01,477 - DEBUG - defaultdict(, {'10.0.0.61': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-12-11 04:17:01,477 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-12-11 04:17:06,596 - INFO - 10.0.0.61 2025-12-11 04:17:06,600 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-12-11 04:17:06,601 - DEBUG - defaultdict(, {'10.0.0.61': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-12-11 04:17:06,601 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-12-11 04:17:11,702 - INFO - 10.0.0.61 2025-12-11 04:17:11,706 - INFO - api-server:active 2025-12-11 04:17:11,707 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.61 2025-12-11 04:17:12,377 - INFO - 10.0.0.33 2025-12-11 04:17:12,381 - INFO - api-server:initializing 2025-12-11 04:17:12,381 - DEBUG - defaultdict(, {'10.0.0.33': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-12-11 04:17:12,381 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-11 04:17:17,490 - INFO - 10.0.0.33 2025-12-11 04:17:17,493 - INFO - api-server:initializing 2025-12-11 04:17:17,493 - DEBUG - defaultdict(, {'10.0.0.33': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-12-11 04:17:17,493 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-12-11 04:17:22,581 - INFO - 10.0.0.33 2025-12-11 04:17:22,587 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-12-11 04:17:22,587 - DEBUG - defaultdict(, {'10.0.0.33': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-12-11 04:17:22,588 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-12-11 04:17:27,688 - INFO - 10.0.0.33 2025-12-11 04:17:27,694 - INFO - api-server:active 2025-12-11 04:17:27,694 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.33 2025-12-11 04:17:28,345 - INFO - 10.0.0.27 2025-12-11 04:17:28,351 - INFO - api-server:initializing 2025-12-11 04:17:28,351 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-12-11 04:17:28,351 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-12-11 04:17:33,451 - INFO - 10.0.0.27 2025-12-11 04:17:33,456 - INFO - api-server:initializing 2025-12-11 04:17:33,456 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-12-11 04:17:33,456 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-12-11 04:17:38,546 - INFO - 10.0.0.27 2025-12-11 04:17:38,551 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-12-11 04:17:38,551 - DEBUG - defaultdict(, {'10.0.0.27': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-12-11 04:17:38,551 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-12-11 04:17:43,642 - INFO - 10.0.0.27 2025-12-11 04:17:43,647 - INFO - api-server:active 2025-12-11 04:17:43,647 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.27 2025-12-11 04:17:43,750 - INFO - 10.0.0.61 2025-12-11 04:17:43,751 - INFO - config 2025-12-11 04:17:43,756 - INFO - config-nodemgr:active 2025-12-11 04:17:43,759 - INFO - api-server:active 2025-12-11 04:17:43,762 - INFO - schema:backup 2025-12-11 04:17:43,764 - INFO - svc-monitor:backup 2025-12-11 04:17:43,767 - INFO - device-manager:active 2025-12-11 04:17:43,768 - INFO - config-database 2025-12-11 04:17:43,768 - INFO - config-cassandra:active 2025-12-11 04:17:43,768 - INFO - config-zookeeper:active 2025-12-11 04:17:43,768 - INFO - config-rabbitmq:active 2025-12-11 04:17:43,768 - INFO - control 2025-12-11 04:17:43,772 - INFO - control-nodemgr:active 2025-12-11 04:17:43,777 - INFO - control:active 2025-12-11 04:17:43,777 - INFO - named:active 2025-12-11 04:17:43,784 - INFO - dns:active 2025-12-11 04:17:43,784 - INFO - analytics 2025-12-11 04:17:43,788 - INFO - analytics-nodemgr:active 2025-12-11 04:17:43,791 - INFO - analytics-api:active 2025-12-11 04:17:43,795 - INFO - collector:active 2025-12-11 04:17:43,795 - INFO - analytics-database 2025-12-11 04:17:43,795 - INFO - analytics-cassandra:active 2025-12-11 04:17:43,800 - INFO - analyticsdb-nodemgr:active 2025-12-11 04:17:43,804 - INFO - query-engine:active 2025-12-11 04:17:43,804 - INFO - webui 2025-12-11 04:17:43,804 - INFO - webui:active 2025-12-11 04:17:43,804 - INFO - webui-middleware:active 2025-12-11 04:17:43,804 - INFO - redis:active 2025-12-11 04:17:43,804 - INFO - analytics_snmp 2025-12-11 04:17:43,808 - INFO - snmp-collector:active 2025-12-11 04:17:43,808 - INFO - snmp-topology:active 2025-12-11 04:17:43,808 - INFO - analytics_alarm 2025-12-11 04:17:43,808 - INFO - alarmgen:active 2025-12-11 04:17:43,899 - INFO - 10.0.0.33 2025-12-11 04:17:43,899 - INFO - config 2025-12-11 04:17:43,905 - INFO - config-nodemgr:active 2025-12-11 04:17:43,909 - INFO - api-server:active 2025-12-11 04:17:43,913 - INFO - schema:active 2025-12-11 04:17:43,918 - INFO - svc-monitor:backup 2025-12-11 04:17:43,921 - INFO - device-manager:backup 2025-12-11 04:17:43,921 - INFO - config-database 2025-12-11 04:17:43,922 - INFO - config-cassandra:active 2025-12-11 04:17:43,922 - INFO - config-zookeeper:active 2025-12-11 04:17:43,922 - INFO - config-rabbitmq:active 2025-12-11 04:17:43,922 - INFO - control 2025-12-11 04:17:43,927 - INFO - control-nodemgr:active 2025-12-11 04:17:43,933 - INFO - control:active 2025-12-11 04:17:43,934 - INFO - named:active 2025-12-11 04:17:43,940 - INFO - dns:active 2025-12-11 04:17:43,941 - INFO - analytics 2025-12-11 04:17:43,946 - INFO - analytics-nodemgr:active 2025-12-11 04:17:43,950 - INFO - analytics-api:active 2025-12-11 04:17:43,955 - INFO - collector:active 2025-12-11 04:17:43,955 - INFO - analytics-database 2025-12-11 04:17:43,955 - INFO - analytics-cassandra:active 2025-12-11 04:17:43,960 - INFO - analyticsdb-nodemgr:active 2025-12-11 04:17:43,965 - INFO - query-engine:active 2025-12-11 04:17:43,965 - INFO - webui 2025-12-11 04:17:43,965 - INFO - webui:active 2025-12-11 04:17:43,965 - INFO - webui-middleware:active 2025-12-11 04:17:43,965 - INFO - redis:active 2025-12-11 04:17:43,965 - INFO - analytics_snmp 2025-12-11 04:17:43,969 - INFO - snmp-collector:active 2025-12-11 04:17:43,969 - INFO - snmp-topology:active 2025-12-11 04:17:43,969 - INFO - analytics_alarm 2025-12-11 04:17:43,970 - INFO - alarmgen:active 2025-12-11 04:17:44,058 - INFO - 10.0.0.27 2025-12-11 04:17:44,058 - INFO - config 2025-12-11 04:17:44,064 - INFO - config-nodemgr:active 2025-12-11 04:17:44,069 - INFO - api-server:active 2025-12-11 04:17:44,072 - INFO - schema:backup 2025-12-11 04:17:44,076 - INFO - svc-monitor:active 2025-12-11 04:17:44,078 - INFO - device-manager:backup 2025-12-11 04:17:44,079 - INFO - config-database 2025-12-11 04:17:44,079 - INFO - config-cassandra:active 2025-12-11 04:17:44,079 - INFO - config-zookeeper:active 2025-12-11 04:17:44,079 - INFO - config-rabbitmq:active 2025-12-11 04:17:44,079 - INFO - control 2025-12-11 04:17:44,083 - INFO - control-nodemgr:active 2025-12-11 04:17:44,088 - INFO - control:active 2025-12-11 04:17:44,088 - INFO - named:active 2025-12-11 04:17:44,093 - INFO - dns:active 2025-12-11 04:17:44,094 - INFO - analytics 2025-12-11 04:17:44,099 - INFO - analytics-nodemgr:active 2025-12-11 04:17:44,105 - INFO - analytics-api:active 2025-12-11 04:17:44,110 - INFO - collector:active 2025-12-11 04:17:44,111 - INFO - analytics-database 2025-12-11 04:17:44,111 - INFO - analytics-cassandra:active 2025-12-11 04:17:44,116 - INFO - analyticsdb-nodemgr:active 2025-12-11 04:17:44,120 - INFO - query-engine:active 2025-12-11 04:17:44,120 - INFO - webui 2025-12-11 04:17:44,121 - INFO - webui:active 2025-12-11 04:17:44,121 - INFO - webui-middleware:active 2025-12-11 04:17:44,121 - INFO - redis:active 2025-12-11 04:17:44,121 - INFO - analytics_snmp 2025-12-11 04:17:44,125 - INFO - snmp-collector:active 2025-12-11 04:17:44,125 - INFO - snmp-topology:active 2025-12-11 04:17:44,125 - INFO - analytics_alarm 2025-12-11 04:17:44,125 - INFO - alarmgen:active 2025-12-11 04:17:44,224 - INFO - 10.0.0.141 2025-12-11 04:17:44,224 - INFO - vrouter 2025-12-11 04:17:44,229 - INFO - vrouter-nodemgr:active 2025-12-11 04:17:44,234 - INFO - agent:active 2025-12-11 04:17:44,323 - INFO - 10.0.0.23 2025-12-11 04:17:44,323 - INFO - vrouter 2025-12-11 04:17:44,328 - INFO - vrouter-nodemgr:active 2025-12-11 04:17:44,333 - INFO - agent:active 2025-12-11 04:17:44,333 - INFO - All the contrail services are up on all nodes 2025-12-11 04:17:44,333 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-1?flat 2025-12-11 04:17:44,709 - INFO - an-jenkins-deploy-platform-ansible-os-4883-1:contrail-vrouter-agent module connection to ['10.20.0.14:5269'] servers UP 2025-12-11 04:17:44,709 - INFO - an-jenkins-deploy-platform-ansible-os-4883-1 connected to xmpp 10.20.0.14 2025-12-11 04:17:44,709 - WARNING - an-jenkins-deploy-platform-ansible-os-4883-1:contrail-vrouter-agent module connection to ['10.20.0.18:5269'] servers NOT UP 2025-12-11 04:17:44,709 - INFO - an-jenkins-deploy-platform-ansible-os-4883-1:contrail-vrouter-agent module connection to ['10.20.0.252:5269'] servers UP 2025-12-11 04:17:44,709 - INFO - an-jenkins-deploy-platform-ansible-os-4883-1 connected to xmpp 10.20.0.252 2025-12-11 04:17:44,709 - INFO - an-jenkins-deploy-platform-ansible-os-4883-1:contrail-vrouter-agent module connection to ['10.20.0.14:53'] servers UP 2025-12-11 04:17:44,709 - INFO - an-jenkins-deploy-platform-ansible-os-4883-1 connected to dns 10.20.0.14 2025-12-11 04:17:44,710 - INFO - an-jenkins-deploy-platform-ansible-os-4883-1:contrail-vrouter-agent module connection to ['10.20.0.18:53'] servers UP 2025-12-11 04:17:44,710 - INFO - an-jenkins-deploy-platform-ansible-os-4883-1 connected to dns 10.20.0.18 2025-12-11 04:17:44,710 - WARNING - an-jenkins-deploy-platform-ansible-os-4883-1:contrail-vrouter-agent module connection to ['10.20.0.252:53'] servers NOT UP 2025-12-11 04:17:44,710 - WARNING - an-jenkins-deploy-platform-ansible-os-4883-1:contrail-vrouter-agent module connection to ['10.0.0.61:8086'] servers NOT UP 2025-12-11 04:17:44,710 - INFO - an-jenkins-deploy-platform-ansible-os-4883-1:contrail-vrouter-agent module connection to ['10.0.0.33:8086'] servers UP 2025-12-11 04:17:44,710 - INFO - an-jenkins-deploy-platform-ansible-os-4883-1 connected to collector 10.0.0.33 2025-12-11 04:17:44,710 - WARNING - an-jenkins-deploy-platform-ansible-os-4883-1:contrail-vrouter-agent module connection to ['10.0.0.27:8086'] servers NOT UP 2025-12-11 04:17:44,710 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4883-1?flat 2025-12-11 04:17:45,114 - INFO - cn-jenkins-deploy-platform-ansible-os-4883-1:contrail-control module connection to 10.0.0.61:9041 servers UP 2025-12-11 04:17:45,114 - INFO - cn-jenkins-deploy-platform-ansible-os-4883-1:contrail-control module connection to 10.0.0.61:5673 servers UP 2025-12-11 04:17:45,114 - INFO - cn-jenkins-deploy-platform-ansible-os-4883-1:contrail-control module connection to ['10.0.0.61:8086'] servers UP 2025-12-11 04:17:45,114 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4883-2?flat 2025-12-11 04:17:45,482 - INFO - cn-jenkins-deploy-platform-ansible-os-4883-2:contrail-control module connection to 10.0.0.61:9041 servers UP 2025-12-11 04:17:45,482 - INFO - cn-jenkins-deploy-platform-ansible-os-4883-2:contrail-control module connection to 10.0.0.61:5673 servers UP 2025-12-11 04:17:45,482 - INFO - cn-jenkins-deploy-platform-ansible-os-4883-2:contrail-control module connection to ['10.0.0.61:8086'] servers UP 2025-12-11 04:17:45,482 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4883-3?flat 2025-12-11 04:17:45,500 - INFO - cn-jenkins-deploy-platform-ansible-os-4883-3:contrail-control module connection to 10.0.0.61:9041 servers UP 2025-12-11 04:17:45,500 - INFO - cn-jenkins-deploy-platform-ansible-os-4883-3:contrail-control module connection to 10.0.0.61:5673 servers UP 2025-12-11 04:17:45,500 - INFO - cn-jenkins-deploy-platform-ansible-os-4883-3:contrail-control module connection to ['10.0.0.61:8086'] servers UP 2025-12-11 04:17:45,500 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4883-1.?flat 2025-12-11 04:17:45,525 - DEBUG - contrail-api is running 2025-12-11 04:17:45,525 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4883-2.?flat 2025-12-11 04:17:45,545 - DEBUG - contrail-api is running 2025-12-11 04:17:45,545 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4883-3.?flat 2025-12-11 04:17:45,566 - DEBUG - contrail-api is running 2025-12-11 04:17:45,828 - INFO - Created VN ctest-vn3-87239494, UUID :f4172fde-e478-4717-b4fa-325ae9204370 2025-12-11 04:17:46,230 - DEBUG - Requesting: http://10.0.0.61:8082/domains 2025-12-11 04:17:46,260 - DEBUG - Requesting: http://10.0.0.61:8082/domain/35136ed0-c5bb-4823-8f5c-ae1b4c17652e 2025-12-11 04:17:46,271 - DEBUG - Requesting: http://10.0.0.61:8082/project/2bc25706-3e1b-4dcd-bbbb-0261d020b882 2025-12-11 04:17:46,394 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/bc21472c-b067-4b88-9e30-60838e877e53 2025-12-11 04:17:46,406 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/bc21472c-b067-4b88-9e30-60838e877e53 2025-12-11 04:17:46,418 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/eda0b80c-f51b-4438-a2ae-ffae70fd182f 2025-12-11 04:17:46,435 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/eda0b80c-f51b-4438-a2ae-ffae70fd182f 2025-12-11 04:17:46,455 - DEBUG - Requesting: http://10.0.0.61:8082/route-target/2c6bea52-5e41-4cf9-b11c-31070197cffb 2025-12-11 04:17:46,461 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-11 04:17:46,461 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/bc21472c-b067-4b88-9e30-60838e877e53 2025-12-11 04:17:46,470 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/eda0b80c-f51b-4438-a2ae-ffae70fd182f 2025-12-11 04:17:46,485 - INFO - Verified VN network id 14 for VN bc21472c-b067-4b88-9e30-60838e877e53 2025-12-11 04:17:46,485 - INFO - Verifications in API Server for VN ctest-vn1-77140010 passed 2025-12-11 04:17:46,486 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/bc21472c-b067-4b88-9e30-60838e877e53 2025-12-11 04:17:46,494 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/eda0b80c-f51b-4438-a2ae-ffae70fd182f 2025-12-11 04:17:46,502 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/eda0b80c-f51b-4438-a2ae-ffae70fd182f 2025-12-11 04:17:46,515 - DEBUG - Requesting: http://10.0.0.61:8082/route-target/2c6bea52-5e41-4cf9-b11c-31070197cffb 2025-12-11 04:17:46,530 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'interests': '1', 'advertised': '1', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-89570916', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13556194610625596296', 'uuid-lslong': '11398716775022952019'}, 'enable': 'true', 'created': '2025-12-11T04:14:51', 'last-modified': '2025-12-11T04:14:52', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:06.547669'} 2025-12-11 04:17:46,537 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-11 04:17:46,545 - DEBUG - Control-node 10.0.0.33 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-89570916', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13556194610625596296', 'uuid-lslong': '11398716775022952019'}, 'enable': 'true', 'created': '2025-12-11T04:14:51', 'last-modified': '2025-12-11T04:14:52', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:01.285943'} 2025-12-11 04:17:46,550 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-11 04:17:46,556 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-89570916', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13556194610625596296', 'uuid-lslong': '11398716775022952019'}, 'enable': 'true', 'created': '2025-12-11T04:14:51', 'last-modified': '2025-12-11T04:14:52', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:55.751911'} 2025-12-11 04:17:46,561 - DEBUG - Route Targets: ['target:64512:8000004'] 2025-12-11 04:17:46,561 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-77140010 passed 2025-12-11 04:17:46,562 - DEBUG - ====Verifying policy data for ctest-vn1-77140010 in API_Server ====== 2025-12-11 04:17:46,562 - DEBUG - Requesting: http://10.0.0.61:8082/domains 2025-12-11 04:17:46,568 - DEBUG - Requesting: http://10.0.0.61:8082/domain/35136ed0-c5bb-4823-8f5c-ae1b4c17652e 2025-12-11 04:17:46,579 - DEBUG - Requesting: http://10.0.0.61:8082/project/2bc25706-3e1b-4dcd-bbbb-0261d020b882 2025-12-11 04:17:46,673 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-network/bc21472c-b067-4b88-9e30-60838e877e53 2025-12-11 04:17:46,683 - DEBUG - ==>Verifying data for policy with id: 1157b7f2-1abd-4ab2-959a-3e7df2ee697d, fqn: ['default-domain', 'ctest-TestBasicVMVN0-89570916', 'policy1'] 2025-12-11 04:17:46,683 - INFO - VN ctest-vn1-77140010 Policy verification: verify_vn_policy_in_api_server, status: True 2025-12-11 04:17:46,683 - DEBUG - Verifying the vn in opserver 2025-12-11 04:17:46,683 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 virtual network link through opserver 10.0.0.61 2025-12-11 04:17:46,683 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-12-11 04:17:46,695 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010?flat'} 2025-12-11 04:17:46,696 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 virtual network link through opserver 10.0.0.33 2025-12-11 04:17:46,696 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-networks 2025-12-11 04:17:46,756 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010?flat'} 2025-12-11 04:17:46,756 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 virtual network link through opserver 10.0.0.27 2025-12-11 04:17:46,757 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-12-11 04:17:46,822 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010?flat'} 2025-12-11 04:17:46,822 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is found in opserver 2025-12-11 04:17:46,835 - DEBUG - VRF ids for VN ctest-vn1-77140010: {'10.0.0.23': '2'} 2025-12-11 04:17:48,323 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4883-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4883-2) 2025-12-11 04:17:48,361 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-12-11 04:17:53,445 - DEBUG - VM is in ACTIVE state now 2025-12-11 04:17:53,445 - INFO - VM name : ctest-vn1_vm2-09089108 2025-12-11 04:17:53,527 - DEBUG - VM ctest-vn1_vm2-09089108 ID is 142bc66e-8a08-4041-88c6-e09e92932075 2025-12-11 04:17:53,603 - DEBUG - VM is in ACTIVE state now 2025-12-11 04:17:53,603 - DEBUG - VM ctest-vn1_vm2-09089108 launched on Node an-jenkins-deploy-platform-ansible-os-4883-2 2025-12-11 04:17:53,688 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075 2025-12-11 04:17:53,725 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075 2025-12-11 04:17:53,734 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/35adc6cb-9aa2-40e7-8951-b85abd507bdb 2025-12-11 04:17:53,745 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075 2025-12-11 04:17:53,755 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/35adc6cb-9aa2-40e7-8951-b85abd507bdb 2025-12-11 04:17:53,766 - DEBUG - Requesting: http://10.0.0.27:8082/instance-ip/390db9be-6192-464f-a827-bfce6d4326dc 2025-12-11 04:17:53,778 - DEBUG - Verifying in api server 10.0.0.27 2025-12-11 04:17:53,778 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075 2025-12-11 04:17:53,795 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/35adc6cb-9aa2-40e7-8951-b85abd507bdb 2025-12-11 04:17:53,805 - DEBUG - Requesting: http://10.0.0.27:8082/instance-ip/390db9be-6192-464f-a827-bfce6d4326dc 2025-12-11 04:17:53,813 - INFO - VM ctest-vn1_vm2-09089108 verfication in all API Servers passed 2025-12-11 04:17:53,835 - DEBUG - VM ctest-vn1_vm2-09089108 Tap interface: {'index': '4', 'name': 'tap35adc6cb-9a', 'uuid': '35adc6cb-9aa2-40e7-8951-b85abd507bdb', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010:ctest-vn1-77140010', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'vm_uuid': '142bc66e-8a08-4041-88c6-e09e92932075', 'vm_name': 'ctest-vn1_vm2-09089108', 'ip_addr': '192.168.1.251', 'mac_addr': '02:35:ad:c6:cb:9a', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-89570916:35adc6cb-9aa2-40e7-8951-b85abd507bdb', 'sg_uuid_list': ['2b857d45-cecf-440b-a7cf-436d4e16a1fb'], 'static_route_list': None, 'vm_project_uuid': '2bc25706-3e1b-4dcd-bbbb-0261d020b882', 'admin_state': 'Enabled', 'flow_key_idx': '34', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-11 04:17:53,835 - DEBUG - Agent 10.0.0.23 vrf name: default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010:ctest-vn1-77140010 2025-12-11 04:17:53,841 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010:ctest-vn1-77140010', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'table_label': '-1', 'vxlan_id': '14', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.241:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-12-11 04:17:53,930 - DEBUG - VM ctest-vn1_vm2-09089108 : Tap interface tap35adc6cb-9a is set to Active 2025-12-11 04:17:53,930 - DEBUG - Tap interface tap35adc6cb-9a detail : {'index': '4', 'name': 'tap35adc6cb-9a', 'uuid': '35adc6cb-9aa2-40e7-8951-b85abd507bdb', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010:ctest-vn1-77140010', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'vm_uuid': '142bc66e-8a08-4041-88c6-e09e92932075', 'vm_name': 'ctest-vn1_vm2-09089108', 'ip_addr': '192.168.1.251', 'mac_addr': '02:35:ad:c6:cb:9a', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-89570916:35adc6cb-9aa2-40e7-8951-b85abd507bdb', 'sg_uuid_list': ['2b857d45-cecf-440b-a7cf-436d4e16a1fb'], 'static_route_list': None, 'vm_project_uuid': '2bc25706-3e1b-4dcd-bbbb-0261d020b882', 'admin_state': 'Enabled', 'flow_key_idx': '34', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-11 04:17:54,045 - DEBUG - Starting Layer 2 verification in Agent 2025-12-11 04:17:54,057 - DEBUG - Layer 2 path is seen for VM MAC 02:35:ad:c6:cb:9a in agent 10.0.0.23 2025-12-11 04:17:54,057 - DEBUG - Active layer 2 route in agent is present for VMI tap35adc6cb-9a 2025-12-11 04:17:54,057 - DEBUG - L2 label(34) matches bw route table and interface table 2025-12-11 04:17:54,072 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is found in Agent of node 10.0.0.23 2025-12-11 04:17:54,203 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is consistent in agent 10.0.0.23 2025-12-11 04:17:54,203 - DEBUG - Route for VM IP ['192.168.1.251'] is consistent in agent 10.0.0.23 2025-12-11 04:17:54,203 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 verification for VM ctest-vn1_vm2-09089108 in Agent 10.0.0.23 passed 2025-12-11 04:17:54,316 - DEBUG - Starting all layer 2 verification in agent 10.0.0.23 2025-12-11 04:17:54,329 - DEBUG - Route for VM MAC 02:35:ad:c6:cb:9a is consistent in agent 10.0.0.23 2025-12-11 04:17:57,532 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1028ms') 2025-12-11 04:17:57,532 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-09089108 failed! 2025-12-11 04:17:57,545 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:18:01,626 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-12-11 04:18:01,626 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-09089108 failed! 2025-12-11 04:18:01,640 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:18:05,721 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-12-11 04:18:05,721 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-09089108 failed! 2025-12-11 04:18:05,734 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:18:09,819 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-12-11 04:18:09,819 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-09089108 failed! 2025-12-11 04:18:09,831 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:18:13,911 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-12-11 04:18:13,911 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-09089108 failed! 2025-12-11 04:18:13,927 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:18:18,009 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-12-11 04:18:18,009 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-09089108 failed! 2025-12-11 04:18:18,024 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:18:22,109 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-12-11 04:18:22,109 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-09089108 failed! 2025-12-11 04:18:22,126 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:18:26,198 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-12-11 04:18:26,198 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-09089108 failed! 2025-12-11 04:18:26,211 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 is 192.168.1.254 and allocation pool is NOT set 2025-12-11 04:18:28,282 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=14.5 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=6.23 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 6.225/10.337/14.450/4.112 ms') 2025-12-11 04:18:28,282 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-09089108 passed 2025-12-11 04:18:28,282 - INFO - VM ctest-vn1_vm2-09089108 verifications in Compute nodes passed 2025-12-11 04:18:28,496 - DEBUG - Validated VM route 192.168.1.251 in vrouter of 10.0.0.23 2025-12-11 04:18:28,500 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-12-11 04:18:28,500 - INFO - Validated routes of VM ctest-vn1_vm2-09089108 in all vrouters 2025-12-11 04:18:28,941 - DEBUG - Starting all layer2 verification in 10.0.0.61 Control Node 2025-12-11 04:18:29,061 - DEBUG - Layer2 route found for VM MAC 02:35:ad:c6:cb:9a in Control-node 10.0.0.61 2025-12-11 04:18:29,061 - DEBUG - L2 Label for VM ctest-vn1_vm2-09089108 same between Control-node 10.0.0.61 and Agent, Expected: 34, Seen: 34 2025-12-11 04:18:29,061 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-12-11 04:18:29,185 - DEBUG - Layer2 route found for VM MAC 02:35:ad:c6:cb:9a in Control-node 10.0.0.27 2025-12-11 04:18:29,185 - DEBUG - L2 Label for VM ctest-vn1_vm2-09089108 same between Control-node 10.0.0.27 and Agent, Expected: 34, Seen: 34 2025-12-11 04:18:29,185 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-09089108 passed 2025-12-11 04:18:29,185 - DEBUG - Verifying the vm in opserver 2025-12-11 04:18:29,185 - DEBUG - Verifying in collector 10.0.0.27 ... 2025-12-11 04:18:29,185 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075?flat 2025-12-11 04:18:29,261 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-89570916:35adc6cb-9aa2-40e7-8951-b85abd507bdb?flat 2025-12-11 04:18:29,279 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-89570916:35adc6cb-9aa2-40e7-8951-b85abd507bdb?flat 2025-12-11 04:18:29,300 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075?flat 2025-12-11 04:18:29,319 - DEBUG - Verifying vm in vn uve 2025-12-11 04:18:29,319 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-89570916:35adc6cb-9aa2-40e7-8951-b85abd507bdb?flat 2025-12-11 04:18:29,338 - DEBUG - VM uve shows interface as {'vm_uuid': '142bc66e-8a08-4041-88c6-e09e92932075', 'ip_address': '192.168.1.251', 'vm_name': 'ctest-vn1_vm2-09089108', 'mac_address': '02:35:ad:c6:cb:9a', 'fixed_ip4_list': ['192.168.1.251'], 'ip6_active': False, 'port_mirror_enabled': False, 'tx_vlan': 65535, 'uuid': '35adc6cb-9aa2-40e7-8951-b85abd507bdb', 'rx_vlan': 65535, 'active': True, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010', 'admin_state': True, 'vhostuser_mode': 0, 'ip6_address': '::', 'vn_uuid': 'bc21472c-b067-4b88-9e30-60838e877e53', 'l2_active': True, 'gateway': '192.168.1.254', '__T': 1765426684397129, 'label': 30, 'is_health_check_active': True, 'ip4_active': True} 2025-12-11 04:18:29,338 - DEBUG - VM uve shows ip address as ['192.168.1.251', '::'] 2025-12-11 04:18:29,338 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 2025-12-11 04:18:29,338 - INFO - Verifying through opserver in 10.0.0.61 2025-12-11 04:18:29,338 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010?flat 2025-12-11 04:18:29,362 - DEBUG - expected vm list 142bc66e-8a08-4041-88c6-e09e92932075 2025-12-11 04:18:29,362 - DEBUG - Extracted vm list ['142bc66e-8a08-4041-88c6-e09e92932075'] 2025-12-11 04:18:29,362 - INFO - VM 142bc66e-8a08-4041-88c6-e09e92932075 is present in default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 2025-12-11 04:18:29,362 - DEBUG - Verifying vm in vrouter uve 2025-12-11 04:18:29,362 - DEBUG - Getting info from collector 10.0.0.27.. 2025-12-11 04:18:29,362 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075?flat 2025-12-11 04:18:29,425 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4883-2'] 2025-12-11 04:18:29,425 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-2:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:18:29,465 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-2?flat 2025-12-11 04:18:29,490 - DEBUG - VM 142bc66e-8a08-4041-88c6-e09e92932075 is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-2 uve 2025-12-11 04:18:29,490 - DEBUG - Validated that VM ctest-vn1_vm2-09089108 is in Vrouter an-jenkins-deploy-platform-ansible-os-4883-2 UVE 2025-12-11 04:18:29,490 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-12-11 04:18:29,497 - DEBUG - Expected tap interface of VM uuid 142bc66e-8a08-4041-88c6-e09e92932075 is default-domain:ctest-TestBasicVMVN0-89570916:35adc6cb-9aa2-40e7-8951-b85abd507bdb 2025-12-11 04:18:29,497 - DEBUG - Expected VN of VM uuid 142bc66e-8a08-4041-88c6-e09e92932075 is default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 2025-12-11 04:18:29,497 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-2:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:18:29,537 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-2?flat 2025-12-11 04:18:29,560 - DEBUG - VM 142bc66e-8a08-4041-88c6-e09e92932075 is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-2 uve 2025-12-11 04:18:29,561 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-89570916:35adc6cb-9aa2-40e7-8951-b85abd507bdb of vm 142bc66e-8a08-4041-88c6-e09e92932075 is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-2 uve 2025-12-11 04:18:29,561 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 of vm 142bc66e-8a08-4041-88c6-e09e92932075 is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-2 uve 2025-12-11 04:18:29,561 - DEBUG - Verifying the 142bc66e-8a08-4041-88c6-e09e92932075 virtual network link through opserver 10.0.0.61 2025-12-11 04:18:29,561 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machines 2025-12-11 04:18:29,571 - DEBUG - VM link and name as {'name': '142bc66e-8a08-4041-88c6-e09e92932075', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075?flat'} 2025-12-11 04:18:29,571 - DEBUG - Verifying the 142bc66e-8a08-4041-88c6-e09e92932075 virtual network link through opserver 10.0.0.33 2025-12-11 04:18:29,571 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-11 04:18:29,631 - DEBUG - VM link and name as {'name': '142bc66e-8a08-4041-88c6-e09e92932075', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075?flat'} 2025-12-11 04:18:29,631 - DEBUG - Verifying the 142bc66e-8a08-4041-88c6-e09e92932075 virtual network link through opserver 10.0.0.27 2025-12-11 04:18:29,631 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-11 04:18:29,644 - DEBUG - VM link and name as {'name': '142bc66e-8a08-4041-88c6-e09e92932075', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075?flat'} 2025-12-11 04:18:29,644 - INFO - VM ctest-vn1_vm2-09089108 validations in Opserver passed 2025-12-11 04:18:30,936 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4883-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4883-1) 2025-12-11 04:18:30,986 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-12-11 04:18:36,077 - DEBUG - VM is in ACTIVE state now 2025-12-11 04:18:36,077 - INFO - VM name : ctest-vn2_vm2-55428441 2025-12-11 04:18:36,153 - DEBUG - VM ctest-vn2_vm2-55428441 ID is 9468a82c-7773-4b4e-b58b-1d348aa1b6fe 2025-12-11 04:18:36,228 - DEBUG - VM is in ACTIVE state now 2025-12-11 04:18:36,228 - DEBUG - VM ctest-vn2_vm2-55428441 launched on Node an-jenkins-deploy-platform-ansible-os-4883-1 2025-12-11 04:18:36,305 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe 2025-12-11 04:18:36,317 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe 2025-12-11 04:18:36,325 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/b1f762de-badc-4501-9bed-da300d46018a 2025-12-11 04:18:36,336 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe 2025-12-11 04:18:36,344 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/b1f762de-badc-4501-9bed-da300d46018a 2025-12-11 04:18:36,356 - DEBUG - Requesting: http://10.0.0.27:8082/instance-ip/48e76345-b3fa-4979-905e-986ad7fbbc9c 2025-12-11 04:18:36,369 - DEBUG - Verifying in api server 10.0.0.27 2025-12-11 04:18:36,369 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe 2025-12-11 04:18:36,378 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/b1f762de-badc-4501-9bed-da300d46018a 2025-12-11 04:18:36,388 - DEBUG - Requesting: http://10.0.0.27:8082/instance-ip/48e76345-b3fa-4979-905e-986ad7fbbc9c 2025-12-11 04:18:36,398 - INFO - VM ctest-vn2_vm2-55428441 verfication in all API Servers passed 2025-12-11 04:18:36,417 - DEBUG - VM ctest-vn2_vm2-55428441 Tap interface: {'index': '4', 'name': 'tapb1f762de-ba', 'uuid': 'b1f762de-badc-4501-9bed-da300d46018a', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119:ctest-vn2-17817119', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119', 'vm_uuid': '9468a82c-7773-4b4e-b58b-1d348aa1b6fe', 'vm_name': 'ctest-vn2_vm2-55428441', 'ip_addr': '192.168.2.251', 'mac_addr': '02:b1:f7:62:de:ba', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '131', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-89570916:b1f762de-badc-4501-9bed-da300d46018a', 'sg_uuid_list': ['2b857d45-cecf-440b-a7cf-436d4e16a1fb'], 'static_route_list': None, 'vm_project_uuid': '2bc25706-3e1b-4dcd-bbbb-0261d020b882', 'admin_state': 'Enabled', 'flow_key_idx': '29', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-11 04:18:36,417 - DEBUG - Agent 10.0.0.141 vrf name: default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119:ctest-vn2-17817119 2025-12-11 04:18:36,423 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119:ctest-vn2-17817119', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.249:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-12-11 04:18:36,512 - DEBUG - VM ctest-vn2_vm2-55428441 : Tap interface tapb1f762de-ba is set to Active 2025-12-11 04:18:36,512 - DEBUG - Tap interface tapb1f762de-ba detail : {'index': '4', 'name': 'tapb1f762de-ba', 'uuid': 'b1f762de-badc-4501-9bed-da300d46018a', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119:ctest-vn2-17817119', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119', 'vm_uuid': '9468a82c-7773-4b4e-b58b-1d348aa1b6fe', 'vm_name': 'ctest-vn2_vm2-55428441', 'ip_addr': '192.168.2.251', 'mac_addr': '02:b1:f7:62:de:ba', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '131', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-89570916:b1f762de-badc-4501-9bed-da300d46018a', 'sg_uuid_list': ['2b857d45-cecf-440b-a7cf-436d4e16a1fb'], 'static_route_list': None, 'vm_project_uuid': '2bc25706-3e1b-4dcd-bbbb-0261d020b882', 'admin_state': 'Enabled', 'flow_key_idx': '29', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-12-11 04:18:36,626 - DEBUG - Starting Layer 2 verification in Agent 2025-12-11 04:18:36,639 - DEBUG - Layer 2 path is seen for VM MAC 02:b1:f7:62:de:ba in agent 10.0.0.141 2025-12-11 04:18:36,640 - DEBUG - Active layer 2 route in agent is present for VMI tapb1f762de-ba 2025-12-11 04:18:36,640 - DEBUG - L2 label(34) matches bw route table and interface table 2025-12-11 04:18:36,647 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 is found in Agent of node 10.0.0.141 2025-12-11 04:18:36,794 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 is consistent in agent 10.0.0.141 2025-12-11 04:18:36,794 - DEBUG - Route for VM IP ['192.168.2.251'] is consistent in agent 10.0.0.141 2025-12-11 04:18:36,794 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 verification for VM ctest-vn2_vm2-55428441 in Agent 10.0.0.141 passed 2025-12-11 04:18:36,905 - DEBUG - Starting all layer 2 verification in agent 10.0.0.141 2025-12-11 04:18:36,925 - DEBUG - Route for VM MAC 02:b1:f7:62:de:ba is consistent in agent 10.0.0.141 2025-12-11 04:18:40,122 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2025-12-11 04:18:40,122 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55428441 failed! 2025-12-11 04:18:40,137 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 is 192.168.2.254 and allocation pool is NOT set 2025-12-11 04:18:44,215 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-12-11 04:18:44,215 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55428441 failed! 2025-12-11 04:18:44,228 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 is 192.168.2.254 and allocation pool is NOT set 2025-12-11 04:18:48,313 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-12-11 04:18:48,313 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55428441 failed! 2025-12-11 04:18:48,326 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 is 192.168.2.254 and allocation pool is NOT set 2025-12-11 04:18:52,405 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-12-11 04:18:52,405 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55428441 failed! 2025-12-11 04:18:52,421 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 is 192.168.2.254 and allocation pool is NOT set 2025-12-11 04:18:56,499 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-12-11 04:18:56,499 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55428441 failed! 2025-12-11 04:18:56,512 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 is 192.168.2.254 and allocation pool is NOT set 2025-12-11 04:19:00,600 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms') 2025-12-11 04:19:00,600 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55428441 failed! 2025-12-11 04:19:00,614 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 is 192.168.2.254 and allocation pool is NOT set 2025-12-11 04:19:04,695 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-12-11 04:19:04,695 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55428441 failed! 2025-12-11 04:19:04,707 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 is 192.168.2.254 and allocation pool is NOT set 2025-12-11 04:19:08,783 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-12-11 04:19:08,784 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55428441 failed! 2025-12-11 04:19:08,799 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 is 192.168.2.254 and allocation pool is NOT set 2025-12-11 04:19:12,888 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms') 2025-12-11 04:19:12,888 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55428441 failed! 2025-12-11 04:19:12,901 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 is 192.168.2.254 and allocation pool is NOT set 2025-12-11 04:19:14,970 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=6.52 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=2.13 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 2.125/4.320/6.516/2.195 ms') 2025-12-11 04:19:14,970 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55428441 passed 2025-12-11 04:19:14,970 - INFO - VM ctest-vn2_vm2-55428441 verifications in Compute nodes passed 2025-12-11 04:19:15,181 - DEBUG - Validated VM route 192.168.2.251 in vrouter of 10.0.0.141 2025-12-11 04:19:15,186 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-12-11 04:19:15,187 - INFO - Validated routes of VM ctest-vn2_vm2-55428441 in all vrouters 2025-12-11 04:19:15,598 - DEBUG - Starting all layer2 verification in 10.0.0.61 Control Node 2025-12-11 04:19:15,733 - DEBUG - Layer2 route found for VM MAC 02:b1:f7:62:de:ba in Control-node 10.0.0.61 2025-12-11 04:19:15,733 - DEBUG - L2 Label for VM ctest-vn2_vm2-55428441 same between Control-node 10.0.0.61 and Agent, Expected: 34, Seen: 34 2025-12-11 04:19:15,733 - DEBUG - Starting all layer2 verification in 10.0.0.27 Control Node 2025-12-11 04:19:15,856 - DEBUG - Layer2 route found for VM MAC 02:b1:f7:62:de:ba in Control-node 10.0.0.27 2025-12-11 04:19:15,856 - DEBUG - L2 Label for VM ctest-vn2_vm2-55428441 same between Control-node 10.0.0.27 and Agent, Expected: 34, Seen: 34 2025-12-11 04:19:15,856 - INFO - Verification in Control-nodes for VM ctest-vn2_vm2-55428441 passed 2025-12-11 04:19:15,856 - DEBUG - Verifying the vm in opserver 2025-12-11 04:19:15,856 - DEBUG - Verifying in collector 10.0.0.27 ... 2025-12-11 04:19:15,856 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe?flat 2025-12-11 04:19:15,872 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-89570916:b1f762de-badc-4501-9bed-da300d46018a?flat 2025-12-11 04:19:15,891 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-89570916:b1f762de-badc-4501-9bed-da300d46018a?flat 2025-12-11 04:19:15,908 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe?flat 2025-12-11 04:19:15,924 - DEBUG - Verifying vm in vn uve 2025-12-11 04:19:15,924 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-89570916:b1f762de-badc-4501-9bed-da300d46018a?flat 2025-12-11 04:19:15,943 - DEBUG - VM uve shows interface as {'port_mirror_enabled': False, '__T': 1765426735890089, 'fixed_ip4_list': ['192.168.2.251'], 'admin_state': True, 'label': 30, 'ip6_active': False, 'rx_vlan': 65535, 'gateway': '192.168.2.254', 'vn_uuid': 'a62f0c86-883a-4ca5-b5bb-5e23d6af5ad1', 'ip4_active': True, 'mac_address': '02:b1:f7:62:de:ba', 'is_health_check_active': True, 'ip6_address': '::', 'ip_address': '192.168.2.251', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119', 'tx_vlan': 65535, 'l2_active': True, 'uuid': 'b1f762de-badc-4501-9bed-da300d46018a', 'active': True, 'vm_name': 'ctest-vn2_vm2-55428441', 'vhostuser_mode': 0, 'vm_uuid': '9468a82c-7773-4b4e-b58b-1d348aa1b6fe'} 2025-12-11 04:19:15,943 - DEBUG - VM uve shows ip address as ['192.168.2.251', '::'] 2025-12-11 04:19:15,943 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 2025-12-11 04:19:15,943 - INFO - Verifying through opserver in 10.0.0.61 2025-12-11 04:19:15,943 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119?flat 2025-12-11 04:19:15,964 - DEBUG - expected vm list 9468a82c-7773-4b4e-b58b-1d348aa1b6fe 2025-12-11 04:19:15,964 - DEBUG - Extracted vm list ['9468a82c-7773-4b4e-b58b-1d348aa1b6fe'] 2025-12-11 04:19:15,964 - INFO - VM 9468a82c-7773-4b4e-b58b-1d348aa1b6fe is present in default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 2025-12-11 04:19:15,964 - DEBUG - Verifying vm in vrouter uve 2025-12-11 04:19:15,964 - DEBUG - Getting info from collector 10.0.0.27.. 2025-12-11 04:19:15,964 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe?flat 2025-12-11 04:19:15,979 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4883-1'] 2025-12-11 04:19:15,979 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-1:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:19:16,022 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-1?flat 2025-12-11 04:19:16,050 - DEBUG - VM 9468a82c-7773-4b4e-b58b-1d348aa1b6fe is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:19:16,050 - DEBUG - Validated that VM ctest-vn2_vm2-55428441 is in Vrouter an-jenkins-deploy-platform-ansible-os-4883-1 UVE 2025-12-11 04:19:16,050 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-12-11 04:19:16,058 - DEBUG - Expected tap interface of VM uuid 9468a82c-7773-4b4e-b58b-1d348aa1b6fe is default-domain:ctest-TestBasicVMVN0-89570916:b1f762de-badc-4501-9bed-da300d46018a 2025-12-11 04:19:16,058 - DEBUG - Expected VN of VM uuid 9468a82c-7773-4b4e-b58b-1d348aa1b6fe is default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 2025-12-11 04:19:16,058 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-1:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:19:16,099 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-1?flat 2025-12-11 04:19:16,126 - DEBUG - VM 9468a82c-7773-4b4e-b58b-1d348aa1b6fe is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:19:16,126 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-89570916:b1f762de-badc-4501-9bed-da300d46018a of vm 9468a82c-7773-4b4e-b58b-1d348aa1b6fe is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:19:16,126 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 of vm 9468a82c-7773-4b4e-b58b-1d348aa1b6fe is present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:19:16,126 - DEBUG - Verifying the 9468a82c-7773-4b4e-b58b-1d348aa1b6fe virtual network link through opserver 10.0.0.61 2025-12-11 04:19:16,126 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machines 2025-12-11 04:19:16,138 - DEBUG - VM link and name as {'name': '9468a82c-7773-4b4e-b58b-1d348aa1b6fe', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe?flat'} 2025-12-11 04:19:16,138 - DEBUG - Verifying the 9468a82c-7773-4b4e-b58b-1d348aa1b6fe virtual network link through opserver 10.0.0.33 2025-12-11 04:19:16,138 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-11 04:19:16,207 - DEBUG - VM link and name as {'name': '9468a82c-7773-4b4e-b58b-1d348aa1b6fe', 'href': 'http://10.0.0.33:8081/analytics/uves/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe?flat'} 2025-12-11 04:19:16,207 - DEBUG - Verifying the 9468a82c-7773-4b4e-b58b-1d348aa1b6fe virtual network link through opserver 10.0.0.27 2025-12-11 04:19:16,207 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-11 04:19:16,220 - DEBUG - VM link and name as {'name': '9468a82c-7773-4b4e-b58b-1d348aa1b6fe', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe?flat'} 2025-12-11 04:19:16,220 - INFO - VM ctest-vn2_vm2-55428441 validations in Opserver passed 2025-12-11 04:19:16,220 - INFO - Waiting for VM ctest-vn1_vm2-09089108 to be up.. 2025-12-11 04:19:16,220 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:19:16,220 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075 2025-12-11 04:19:16,227 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/35adc6cb-9aa2-40e7-8951-b85abd507bdb 2025-12-11 04:19:17,388 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.51 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.559 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.559/2.036/3.514/1.477 ms') 2025-12-11 04:19:17,388 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-09089108 passed 2025-12-11 04:19:17,462 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:19:17,462 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-09089108, IP 192.168.1.251, Port 22 2025-12-11 04:19:17,637 - DEBUG - VM ctest-vn1_vm2-09089108 is ready for SSH connections 2025-12-11 04:19:17,637 - INFO - Waiting for VM ctest-vn2_vm2-55428441 to be up.. 2025-12-11 04:19:17,637 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:19:17,638 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe 2025-12-11 04:19:17,649 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine-interface/b1f762de-badc-4501-9bed-da300d46018a 2025-12-11 04:19:18,823 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=4.49 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.547 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.547/2.517/4.487/1.970 ms') 2025-12-11 04:19:18,824 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-55428441 passed 2025-12-11 04:19:18,902 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:19:18,902 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-55428441, IP 192.168.2.251, Port 22 2025-12-11 04:19:18,968 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-11 04:19:19,052 - DEBUG - VM ctest-vn2_vm2-55428441 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-11 04:19:24,052 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:19:24,052 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-55428441, IP 192.168.2.251, Port 22 2025-12-11 04:19:24,120 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-11 04:19:24,205 - DEBUG - VM ctest-vn2_vm2-55428441 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-11 04:19:29,206 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:19:29,206 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-55428441, IP 192.168.2.251, Port 22 2025-12-11 04:19:29,272 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-12-11 04:19:29,346 - DEBUG - VM ctest-vn2_vm2-55428441 is NOT ready for SSH connections, VM status: ACTIVE 2025-12-11 04:19:34,347 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:19:34,347 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-55428441, IP 192.168.2.251, Port 22 2025-12-11 04:19:34,516 - DEBUG - VM ctest-vn2_vm2-55428441 is ready for SSH connections 2025-12-11 04:19:34,517 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-12-11 04:19:34,517 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.251, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.23, gateway password: c0ntrail123 2025-12-11 04:19:34,517 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.251 2025-12-11 04:20:15,273 - DEBUG - PING 192.168.2.251 (192.168.2.251) 56(84) bytes of data. 64 bytes from 192.168.2.251: icmp_req=1 ttl=63 time=11.1 ms 64 bytes from 192.168.2.251: icmp_req=2 ttl=63 time=1.47 ms 64 bytes from 192.168.2.251: icmp_req=3 ttl=63 time=1.03 ms --- 192.168.2.251 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.039/4.544/11.121/4.654 ms 2025-12-11 04:20:15,273 - INFO - Ping to IP 192.168.2.251 from VM ctest-vn1_vm2-09089108 passed 2025-12-11 04:20:15,273 - INFO - Deleting VM ctest-vn2_vm2-55428441 2025-12-11 04:20:15,349 - DEBUG - Verifying in api server 10.0.0.27 2025-12-11 04:20:15,349 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe 2025-12-11 04:20:15,357 - DEBUG - VM ID 9468a82c-7773-4b4e-b58b-1d348aa1b6fe of VM ctest-vn2_vm2-55428441 is still found in API Server 2025-12-11 04:20:17,357 - DEBUG - Verifying in api server 10.0.0.27 2025-12-11 04:20:17,357 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe 2025-12-11 04:20:17,363 - DEBUG - Response Code: 404 2025-12-11 04:20:17,363 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe 2025-12-11 04:20:17,415 - DEBUG - Response Code: 404 2025-12-11 04:20:17,415 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/9468a82c-7773-4b4e-b58b-1d348aa1b6fe 2025-12-11 04:20:17,420 - DEBUG - Response Code: 404 2025-12-11 04:20:17,420 - INFO - VM ctest-vn2_vm2-55428441 is fully removed in API-Server 2025-12-11 04:20:17,572 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-12-11 04:20:17,732 - DEBUG - VM route 192.168.2.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 of compute 10.0.0.141 2025-12-11 04:20:17,737 - DEBUG - Validated that vrouter 10.0.0.141 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 2025-12-11 04:20:17,737 - DEBUG - Validated that vrouter 10.0.0.23 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 2025-12-11 04:20:17,737 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 2025-12-11 04:20:17,737 - INFO - VM ctest-vn2_vm2-55428441 is removed in Compute, and routes are removed in all compute nodes 2025-12-11 04:20:17,948 - INFO - Routes for VM ctest-vn2_vm2-55428441 is removed in all control-nodes 2025-12-11 04:20:17,948 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-1:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:20:17,993 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-1?flat 2025-12-11 04:20:18,020 - DEBUG - VM 9468a82c-7773-4b4e-b58b-1d348aa1b6fe is still present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:20:18,020 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-11 04:20:18,020 - DEBUG - interface for vm 9468a82c-7773-4b4e-b58b-1d348aa1b6fe deleted from vrouter uve ... 2025-12-11 04:20:18,020 - DEBUG - Deleted interface not in error interface list ... 2025-12-11 04:20:21,020 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-1:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:20:21,064 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-1?flat 2025-12-11 04:20:21,092 - DEBUG - VM 9468a82c-7773-4b4e-b58b-1d348aa1b6fe is still present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:20:21,092 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-11 04:20:21,092 - DEBUG - interface for vm 9468a82c-7773-4b4e-b58b-1d348aa1b6fe deleted from vrouter uve ... 2025-12-11 04:20:21,092 - DEBUG - Deleted interface not in error interface list ... 2025-12-11 04:20:24,093 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-1:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:20:24,136 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-1?flat 2025-12-11 04:20:24,172 - DEBUG - VM 9468a82c-7773-4b4e-b58b-1d348aa1b6fe is still present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:20:24,172 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-11 04:20:24,172 - DEBUG - interface for vm 9468a82c-7773-4b4e-b58b-1d348aa1b6fe deleted from vrouter uve ... 2025-12-11 04:20:24,172 - DEBUG - Deleted interface not in error interface list ... 2025-12-11 04:20:27,173 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-1:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:20:27,222 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-1?flat 2025-12-11 04:20:27,250 - DEBUG - VM 9468a82c-7773-4b4e-b58b-1d348aa1b6fe is not present in vrouter an-jenkins-deploy-platform-ansible-os-4883-1 uve 2025-12-11 04:20:27,250 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-11 04:20:27,250 - DEBUG - interface for vm 9468a82c-7773-4b4e-b58b-1d348aa1b6fe deleted from vrouter uve ... 2025-12-11 04:20:27,250 - DEBUG - Deleted interface not in error interface list ... 2025-12-11 04:20:27,250 - DEBUG - Validated that VM 9468a82c-7773-4b4e-b58b-1d348aa1b6fe is removed in Vrouter UVE 2025-12-11 04:20:27,250 - DEBUG - Verifying through opserver in 10.0.0.61 2025-12-11 04:20:27,250 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119?flat 2025-12-11 04:20:27,272 - DEBUG - 9468a82c-7773-4b4e-b58b-1d348aa1b6fe BM not in default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn2-17817119 uve 2025-12-11 04:20:27,272 - DEBUG - Validated that VM 9468a82c-7773-4b4e-b58b-1d348aa1b6fe is not present in the VN UVE 2025-12-11 04:20:27,272 - DEBUG - Verifying the 9468a82c-7773-4b4e-b58b-1d348aa1b6fe virtual network link through opserver 10.0.0.61 2025-12-11 04:20:27,272 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machines 2025-12-11 04:20:27,285 - INFO - 9468a82c-7773-4b4e-b58b-1d348aa1b6fe vm uve deleted from opserver 2025-12-11 04:20:27,285 - DEBUG - Verifying the 9468a82c-7773-4b4e-b58b-1d348aa1b6fe virtual network link through opserver 10.0.0.33 2025-12-11 04:20:27,285 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-11 04:20:27,297 - INFO - 9468a82c-7773-4b4e-b58b-1d348aa1b6fe vm uve deleted from opserver 2025-12-11 04:20:27,297 - DEBUG - Verifying the 9468a82c-7773-4b4e-b58b-1d348aa1b6fe virtual network link through opserver 10.0.0.27 2025-12-11 04:20:27,297 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-11 04:20:27,309 - INFO - 9468a82c-7773-4b4e-b58b-1d348aa1b6fe vm uve deleted from opserver 2025-12-11 04:20:27,309 - INFO - Validated that VM 9468a82c-7773-4b4e-b58b-1d348aa1b6fe is removed from Opserver 2025-12-11 04:20:27,309 - INFO - Deleting VM ctest-vn1_vm2-09089108 2025-12-11 04:20:27,378 - DEBUG - Verifying in api server 10.0.0.27 2025-12-11 04:20:27,378 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075 2025-12-11 04:20:27,387 - DEBUG - VM ID 142bc66e-8a08-4041-88c6-e09e92932075 of VM ctest-vn1_vm2-09089108 is still found in API Server 2025-12-11 04:20:29,387 - DEBUG - Verifying in api server 10.0.0.27 2025-12-11 04:20:29,387 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075 2025-12-11 04:20:29,395 - DEBUG - Response Code: 404 2025-12-11 04:20:29,395 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075 2025-12-11 04:20:29,401 - DEBUG - Response Code: 404 2025-12-11 04:20:29,402 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-machine/142bc66e-8a08-4041-88c6-e09e92932075 2025-12-11 04:20:29,407 - DEBUG - Response Code: 404 2025-12-11 04:20:29,407 - INFO - VM ctest-vn1_vm2-09089108 is fully removed in API-Server 2025-12-11 04:20:29,564 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-12-11 04:20:29,598 - DEBUG - Validated that vrouter 10.0.0.141 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 2025-12-11 04:20:29,733 - DEBUG - VM route 192.168.1.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 of compute 10.0.0.23 2025-12-11 04:20:29,737 - DEBUG - Validated that vrouter 10.0.0.23 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 2025-12-11 04:20:29,737 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 2025-12-11 04:20:29,737 - INFO - VM ctest-vn1_vm2-09089108 is removed in Compute, and routes are removed in all compute nodes 2025-12-11 04:20:29,935 - INFO - Routes for VM ctest-vn1_vm2-09089108 is removed in all control-nodes 2025-12-11 04:20:29,936 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-2:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:20:29,988 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-2?flat 2025-12-11 04:20:30,015 - DEBUG - VM 142bc66e-8a08-4041-88c6-e09e92932075 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4883-2 uve 2025-12-11 04:20:30,015 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-11 04:20:30,015 - DEBUG - interface for vm 142bc66e-8a08-4041-88c6-e09e92932075 deleted from vrouter uve ... 2025-12-11 04:20:30,015 - DEBUG - Deleted interface not in error interface list ... 2025-12-11 04:20:33,016 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-2:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:20:33,058 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-2?flat 2025-12-11 04:20:33,084 - DEBUG - VM 142bc66e-8a08-4041-88c6-e09e92932075 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4883-2 uve 2025-12-11 04:20:33,084 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-11 04:20:33,085 - DEBUG - interface for vm 142bc66e-8a08-4041-88c6-e09e92932075 deleted from vrouter uve ... 2025-12-11 04:20:33,085 - DEBUG - Deleted interface not in error interface list ... 2025-12-11 04:20:36,085 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4883-2:Compute:contrail-vrouter-agent:0?flat 2025-12-11 04:20:36,128 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4883-2?flat 2025-12-11 04:20:36,154 - DEBUG - VM 142bc66e-8a08-4041-88c6-e09e92932075 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4883-2 uve 2025-12-11 04:20:36,154 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-12-11 04:20:36,154 - DEBUG - interface for vm 142bc66e-8a08-4041-88c6-e09e92932075 deleted from vrouter uve ... 2025-12-11 04:20:36,154 - DEBUG - Deleted interface not in error interface list ... 2025-12-11 04:20:36,154 - DEBUG - Validated that VM 142bc66e-8a08-4041-88c6-e09e92932075 is removed in Vrouter UVE 2025-12-11 04:20:36,154 - DEBUG - Verifying through opserver in 10.0.0.61 2025-12-11 04:20:36,155 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010?flat 2025-12-11 04:20:36,175 - DEBUG - 142bc66e-8a08-4041-88c6-e09e92932075 BM not in default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 uve 2025-12-11 04:20:36,175 - DEBUG - Validated that VM 142bc66e-8a08-4041-88c6-e09e92932075 is not present in the VN UVE 2025-12-11 04:20:36,175 - DEBUG - Verifying the 142bc66e-8a08-4041-88c6-e09e92932075 virtual network link through opserver 10.0.0.61 2025-12-11 04:20:36,175 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-machines 2025-12-11 04:20:36,189 - INFO - 142bc66e-8a08-4041-88c6-e09e92932075 vm uve deleted from opserver 2025-12-11 04:20:36,189 - DEBUG - Verifying the 142bc66e-8a08-4041-88c6-e09e92932075 virtual network link through opserver 10.0.0.33 2025-12-11 04:20:36,189 - DEBUG - Requesting: http://10.0.0.33:8081/analytics/uves/virtual-machines 2025-12-11 04:20:36,203 - INFO - 142bc66e-8a08-4041-88c6-e09e92932075 vm uve deleted from opserver 2025-12-11 04:20:36,203 - DEBUG - Verifying the 142bc66e-8a08-4041-88c6-e09e92932075 virtual network link through opserver 10.0.0.27 2025-12-11 04:20:36,203 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-machines 2025-12-11 04:20:36,215 - INFO - 142bc66e-8a08-4041-88c6-e09e92932075 vm uve deleted from opserver 2025-12-11 04:20:36,215 - INFO - Validated that VM 142bc66e-8a08-4041-88c6-e09e92932075 is removed from Opserver 2025-12-11 04:20:36,215 - INFO - Deleting VN ctest-vn3-87239494 2025-12-11 04:20:36,317 - INFO - Deleting VM ctest-vn2_vm1-34612927 2025-12-11 04:20:36,399 - INFO - Deleting VM ctest-vn1_vm1-43073363 2025-12-11 04:20:36,608 - INFO - Deleting VN ctest-vn2-17817119 2025-12-11 04:20:36,664 - DEBUG - RefsExistError Delete when resource still referred: ['http://10.0.0.27:8082/virtual-machine-interface/ca815bb4-a182-404c-9da1-e38ccc8bb731'] while deleting VN ctest-vn2-17817119..Will retry 2025-12-11 04:20:38,760 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/bc35afab-d0bc-4963-98b0-ed3d87b5c12d 2025-12-11 04:20:38,765 - DEBUG - Response Code: 404 2025-12-11 04:20:38,765 - DEBUG - Requesting: http://10.0.0.61:8082/domains 2025-12-11 04:20:38,771 - DEBUG - Requesting: http://10.0.0.61:8082/domain/35136ed0-c5bb-4823-8f5c-ae1b4c17652e 2025-12-11 04:20:38,788 - DEBUG - Requesting: http://10.0.0.61:8082/project/2bc25706-3e1b-4dcd-bbbb-0261d020b882 2025-12-11 04:20:38,880 - INFO - Validated that VN ctest-vn2-17817119 is not found in API Server 2025-12-11 04:20:38,891 - DEBUG - VN ctest-vn2-17817119 is not present in Agent 10.0.0.141 2025-12-11 04:20:38,902 - DEBUG - VN ctest-vn2-17817119 is not present in Agent 10.0.0.23 2025-12-11 04:20:38,902 - INFO - Validated that VN ctest-vn2-17817119 is not in any agent 2025-12-11 04:20:38,931 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-17817119 info 2025-12-11 04:20:39,096 - INFO - Deleting VN ctest-vn1-77140010 2025-12-11 04:20:39,204 - DEBUG - Requesting: http://10.0.0.61:8082/routing-instance/eda0b80c-f51b-4438-a2ae-ffae70fd182f 2025-12-11 04:20:39,209 - DEBUG - Response Code: 404 2025-12-11 04:20:39,210 - DEBUG - Requesting: http://10.0.0.61:8082/domains 2025-12-11 04:20:39,214 - DEBUG - Requesting: http://10.0.0.61:8082/domain/35136ed0-c5bb-4823-8f5c-ae1b4c17652e 2025-12-11 04:20:39,236 - DEBUG - Requesting: http://10.0.0.61:8082/project/2bc25706-3e1b-4dcd-bbbb-0261d020b882 2025-12-11 04:20:39,322 - INFO - Validated that VN ctest-vn1-77140010 is not found in API Server 2025-12-11 04:20:39,332 - DEBUG - VN ctest-vn1-77140010 is not present in Agent 10.0.0.141 2025-12-11 04:20:39,342 - DEBUG - VN ctest-vn1-77140010 is not present in Agent 10.0.0.23 2025-12-11 04:20:39,342 - INFO - Validated that VN ctest-vn1-77140010 is not in any agent 2025-12-11 04:20:39,347 - DEBUG - VRF 2 is not seen in agent 10.0.0.23 2025-12-11 04:20:39,352 - DEBUG - Vrouter 10.0.0.23 does not have vrf 2 for VN ctest-vn1-77140010 2025-12-11 04:20:39,352 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestBasicVMVN0-89570916:ctest-vn1-77140010 2025-12-11 04:20:39,371 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-77140010 info 2025-12-11 04:20:39,408 - INFO - Deleted policy policy2 2025-12-11 04:20:39,465 - INFO - Deleted policy policy1 2025-12-11 04:20:40,718 - DEBUG - Skipping xmpp flap check 2025-12-11 04:20:40,718 - INFO - END TEST : test_process_restart_in_policy_between_vns : PASSED[0:05:51] 2025-12-11 04:20:40,719 - INFO - -------------------------------------------------------------------------------- 2025-12-11 04:20:40,721 - INFO - ================================================================================ 2025-12-11 04:20:40,721 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling 2025-12-11 04:20:40,721 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229). 1. Send broadcast traffic from one compute node. 2. Other compute in same subnet should receive that traffic. 3. Receiving compute should treat this traffic as underlay. 4. Compute should not replicate the packet and send the copy back. Pass criteria: Step 3-4 should pass Maintainer : chhandak@juniper.net 2025-12-11 04:20:41,979 - DEBUG - Skipping xmpp flap check 2025-12-11 04:20:41,979 - INFO - Initial checks done. Running the testcase now 2025-12-11 04:20:41,980 - INFO - 2025-12-11 04:20:42,367 - INFO - Agent interface name: ens6 2025-12-11 04:20:42,460 - DEBUG - Executing command: sudo tcpdump -nni ens6 -U host 10.20.0.255 -w /tmp/ens6_ctest-random-23834457.pcap 2025-12-11 04:20:47,866 - INFO - Pinging broacast address 10.20.0.255 from compute 10.20.0.249 2025-12-11 04:21:02,164 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-12-11 04:21:04,317 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-23834457.pcap | grep -c length 2025-12-11 04:21:04,332 - DEBUG - STDOUT: 10 2025-12-11 04:21:04,332 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-23834457.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-12-11 04:21:04,332 - INFO - 10 packets are found in tcpdump output as expected 2025-12-11 04:21:04,332 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-12-11 04:21:06,377 - INFO - Packet count matched: Compute 10.0.0.23 has receive only 10 packet from source IP 10.20.0.255. No duplicate packet seen 2025-12-11 04:21:07,643 - DEBUG - Skipping xmpp flap check 2025-12-11 04:21:07,643 - INFO - END TEST : test_underlay_broadcast_traffic_handling : PASSED[0:00:27] 2025-12-11 04:21:07,643 - INFO - -------------------------------------------------------------------------------- 2025-12-11 04:21:08,402 - INFO - Deleted project: ctest-TestBasicVMVN0-89570916, ID : 2bc25706-3e1b-4dcd-bbbb-0261d020b882