2025-08-28 03:58:31,611 - INFO - Domain Default found not creating 2025-08-28 03:58:31,795 - INFO - Project ctest-TestBasicVMVN0-72692947 not found, creating it 2025-08-28 03:58:32,335 - INFO - Created Project:ctest-TestBasicVMVN0-72692947, ID : 5631e639-3d56-4dbe-9639-2fd2c3fa5ac2 2025-08-28 03:58:33,927 - INFO - ================================================================================ 2025-08-28 03:58:33,928 - INFO - STARTING TEST : test_control_node_switchover 2025-08-28 03:58:33,928 - INFO - TEST DESCRIPTION : Stop the control node and check peering with agent fallback to other control node. 1. Pick one VN from respource pool which has 2 VM's in it 2. Verify ping between VM's 3. Find active control node in cluster by agent inspect 4. Stop control service on active control node 5. Verify agents are connected to new active control-node using xmpp connections 6. Bring back control service on previous active node 7. Verify ping between VM's again after bringing up control serveice Pass criteria: Step 2,5 and 7 should pass 2025-08-28 03:58:35,190 - DEBUG - Skipping xmpp flap check 2025-08-28 03:58:35,190 - INFO - Initial checks done. Running the testcase now 2025-08-28 03:58:35,190 - INFO - 2025-08-28 03:58:35,939 - DEBUG - Response for create_network : {'network': {'id': '6c1038b4-167a-42ef-a098-3298511148db', 'name': 'ctest-vn1-29522296', 'tenant_id': '5631e6393d564dbe96392fd2c3fa5ac2', 'project_id': '5631e6393d564dbe96392fd2c3fa5ac2', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicVMVN0-72692947', 'ctest-vn1-29522296'], 'port_security_enabled': True, 'description': ''}} 2025-08-28 03:58:36,246 - DEBUG - Response for create_subnet : {'subnet': {'id': '8d58ff05-aed8-4356-b046-247779e11744', 'name': '', 'tenant_id': '5631e6393d564dbe96392fd2c3fa5ac2', 'network_id': '6c1038b4-167a-42ef-a098-3298511148db', 'ip_version': 4, 'cidr': '192.168.1.0/24', 'allocation_pools': [{'start': '192.168.1.2', 'end': '192.168.1.254'}], 'gateway_ip': '192.168.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '192.168.1.2', 'tags': [], 'project_id': '5631e6393d564dbe96392fd2c3fa5ac2'}} 2025-08-28 03:58:36,290 - INFO - Created VN ctest-vn1-29522296 2025-08-28 03:58:36,314 - DEBUG - VN ctest-vn1-29522296 UUID is 6c1038b4-167a-42ef-a098-3298511148db 2025-08-28 03:58:36,418 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 03:58:36,720 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 03:58:36,750 - DEBUG - Requesting: http://10.0.0.19:8082/domain/692c5a68-a199-4232-a07c-1cd4918de9d4 2025-08-28 03:58:36,767 - DEBUG - Requesting: http://10.0.0.19:8082/project/5631e639-3d56-4dbe-9639-2fd2c3fa5ac2 2025-08-28 03:58:36,891 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/6c1038b4-167a-42ef-a098-3298511148db 2025-08-28 03:58:36,906 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/6c1038b4-167a-42ef-a098-3298511148db 2025-08-28 03:58:36,921 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/4ee685b5-7946-48da-b6df-484d200631ec 2025-08-28 03:58:36,933 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/4ee685b5-7946-48da-b6df-484d200631ec 2025-08-28 03:58:36,947 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/019b3c21-89b1-4a47-8e7a-5a8f2164a068 2025-08-28 03:58:36,956 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 03:58:36,956 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/6c1038b4-167a-42ef-a098-3298511148db 2025-08-28 03:58:36,971 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/4ee685b5-7946-48da-b6df-484d200631ec 2025-08-28 03:58:37,001 - INFO - Verified VN network id 12 for VN 6c1038b4-167a-42ef-a098-3298511148db 2025-08-28 03:58:37,001 - INFO - Verifications in API Server for VN ctest-vn1-29522296 passed 2025-08-28 03:58:37,002 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/6c1038b4-167a-42ef-a098-3298511148db 2025-08-28 03:58:37,012 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/4ee685b5-7946-48da-b6df-484d200631ec 2025-08-28 03:58:37,021 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/4ee685b5-7946-48da-b6df-484d200631ec 2025-08-28 03:58:37,031 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/019b3c21-89b1-4a47-8e7a-5a8f2164a068 2025-08-28 03:58:37,045 - DEBUG - Control-node 10.0.0.19 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-72692947', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7786786101845967599', 'uuid-lslong': '11572054872304994523'}, 'enable': 'true', 'created': '2025-08-28T03:58:35', 'last-modified': '2025-08-28T03:58:36', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.875073'} 2025-08-28 03:58:37,060 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 03:58:37,069 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-72692947', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7786786101845967599', 'uuid-lslong': '11572054872304994523'}, 'enable': 'true', 'created': '2025-08-28T03:58:35', 'last-modified': '2025-08-28T03:58:36', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.892501'} 2025-08-28 03:58:37,078 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 03:58:37,084 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-72692947', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7786786101845967599', 'uuid-lslong': '11572054872304994523'}, 'enable': 'true', 'created': '2025-08-28T03:58:35', 'last-modified': '2025-08-28T03:58:36', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.920582'} 2025-08-28 03:58:37,093 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 03:58:37,093 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-29522296 passed 2025-08-28 03:58:37,093 - DEBUG - ====Verifying policy data for ctest-vn1-29522296 in API_Server ====== 2025-08-28 03:58:37,094 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 03:58:37,101 - DEBUG - Requesting: http://10.0.0.19:8082/domain/692c5a68-a199-4232-a07c-1cd4918de9d4 2025-08-28 03:58:37,115 - DEBUG - Requesting: http://10.0.0.19:8082/project/5631e639-3d56-4dbe-9639-2fd2c3fa5ac2 2025-08-28 03:58:37,235 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/6c1038b4-167a-42ef-a098-3298511148db 2025-08-28 03:58:37,245 - DEBUG - =>VN ctest-vn1-29522296 has no policy to be verified 2025-08-28 03:58:37,245 - DEBUG - Verifying the vn in opserver 2025-08-28 03:58:37,245 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 virtual network link through opserver 10.0.0.19 2025-08-28 03:58:37,245 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2025-08-28 03:58:37,583 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2025-08-28 03:58:37,645 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296?flat'} 2025-08-28 03:58:37,645 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 virtual network link through opserver 10.0.0.38 2025-08-28 03:58:37,645 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-28 03:58:37,960 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-28 03:58:38,024 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296?flat'} 2025-08-28 03:58:38,024 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 virtual network link through opserver 10.0.0.41 2025-08-28 03:58:38,024 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-08-28 03:58:38,348 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-08-28 03:58:38,401 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296?flat'} 2025-08-28 03:58:38,401 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is found in opserver 2025-08-28 03:58:38,422 - DEBUG - Do not have enough data to verify VN in agent 2025-08-28 03:58:38,433 - DEBUG - VRF ids for VN ctest-vn1-29522296: {} 2025-08-28 03:58:38,484 - DEBUG - Services list from nova: [, , , ] 2025-08-28 03:58:39,478 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3956-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3956-1) 2025-08-28 03:58:40,814 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3956-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3956-2) 2025-08-28 03:58:40,814 - INFO - Waiting for VM ctest-vn1_vm1-60404630 to be up.. 2025-08-28 03:58:40,903 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-08-28 03:58:45,988 - DEBUG - VM is in ACTIVE state now 2025-08-28 03:58:45,988 - INFO - VM name : ctest-vn1_vm1-60404630 2025-08-28 03:58:46,066 - DEBUG - VM ctest-vn1_vm1-60404630 ID is badab9b2-b145-4df1-b61e-027fb2e759e5 2025-08-28 03:58:46,089 - DEBUG - VM ctest-vn1_vm1-60404630 launched on Node an-jenkins-deploy-platform-ansible-os-3956-1 2025-08-28 03:58:46,169 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5 2025-08-28 03:58:46,475 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5 2025-08-28 03:58:46,506 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/7116d704-0c73-45ec-a73e-0ed717b95a99 2025-08-28 03:58:49,680 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-08-28 03:58:49,680 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-60404630 failed! 2025-08-28 03:58:49,694 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is 192.168.1.1 and allocation pool is NOT set 2025-08-28 03:58:53,778 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-08-28 03:58:53,778 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-60404630 failed! 2025-08-28 03:58:53,790 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is 192.168.1.1 and allocation pool is NOT set 2025-08-28 03:58:57,871 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-08-28 03:58:57,871 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-60404630 failed! 2025-08-28 03:58:57,884 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is 192.168.1.1 and allocation pool is NOT set 2025-08-28 03:59:01,971 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1024ms') 2025-08-28 03:59:01,972 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-60404630 failed! 2025-08-28 03:59:01,991 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is 192.168.1.1 and allocation pool is NOT set 2025-08-28 03:59:06,059 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms') 2025-08-28 03:59:06,060 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-60404630 failed! 2025-08-28 03:59:06,074 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is 192.168.1.1 and allocation pool is NOT set 2025-08-28 03:59:10,164 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms') 2025-08-28 03:59:10,164 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-60404630 failed! 2025-08-28 03:59:10,178 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is 192.168.1.1 and allocation pool is NOT set 2025-08-28 03:59:14,257 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms') 2025-08-28 03:59:14,257 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-60404630 failed! 2025-08-28 03:59:14,271 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is 192.168.1.1 and allocation pool is NOT set 2025-08-28 03:59:16,331 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=8.31 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.85 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.848/5.077/8.306/3.229 ms') 2025-08-28 03:59:16,331 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-60404630 passed 2025-08-28 03:59:16,398 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 03:59:16,398 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-60404630, IP 192.168.1.3, Port 22 2025-08-28 03:59:16,465 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-28 03:59:16,543 - DEBUG - VM ctest-vn1_vm1-60404630 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-28 03:59:21,544 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 03:59:21,544 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-60404630, IP 192.168.1.3, Port 22 2025-08-28 03:59:21,603 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-28 03:59:21,682 - DEBUG - VM ctest-vn1_vm1-60404630 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-28 03:59:26,683 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 03:59:26,683 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-60404630, IP 192.168.1.3, Port 22 2025-08-28 03:59:26,759 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-28 03:59:26,840 - DEBUG - VM ctest-vn1_vm1-60404630 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-28 03:59:31,840 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 03:59:31,840 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-60404630, IP 192.168.1.3, Port 22 2025-08-28 03:59:31,910 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-28 03:59:31,991 - DEBUG - VM ctest-vn1_vm1-60404630 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-28 03:59:36,992 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 03:59:36,992 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-60404630, IP 192.168.1.3, Port 22 2025-08-28 03:59:37,158 - DEBUG - VM ctest-vn1_vm1-60404630 is ready for SSH connections 2025-08-28 03:59:37,158 - INFO - Waiting for VM ctest-vn1_vm2-98134178 to be up.. 2025-08-28 03:59:37,279 - DEBUG - VM is in ACTIVE state now 2025-08-28 03:59:37,279 - INFO - VM name : ctest-vn1_vm2-98134178 2025-08-28 03:59:37,366 - DEBUG - VM ctest-vn1_vm2-98134178 ID is de59cee3-9395-4745-bd10-ef98d546c004 2025-08-28 03:59:37,366 - DEBUG - VM ctest-vn1_vm2-98134178 launched on Node an-jenkins-deploy-platform-ansible-os-3956-2 2025-08-28 03:59:37,462 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de59cee3-9395-4745-bd10-ef98d546c004 2025-08-28 03:59:37,476 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/57101bdd-2a06-416f-9478-a6341fe66b2c 2025-08-28 03:59:38,639 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=2.60 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=2.68 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 2.596/2.637/2.678/0.041 ms') 2025-08-28 03:59:38,639 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-98134178 passed 2025-08-28 03:59:38,721 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 03:59:38,721 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-98134178, IP 192.168.1.4, Port 22 2025-08-28 03:59:38,887 - DEBUG - VM ctest-vn1_vm2-98134178 is ready for SSH connections 2025-08-28 03:59:38,888 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 03:59:38,888 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.65, gateway password: c0ntrail123 2025-08-28 03:59:38,888 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-08-28 03:59:57,757 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=11.1 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.40 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.09 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.094/4.561/11.191/4.689 ms 2025-08-28 03:59:57,757 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-60404630 passed 2025-08-28 03:59:57,757 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 03:59:57,757 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-08-28 03:59:57,757 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-08-28 04:00:32,448 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=5.57 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.25 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.26 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.255/2.698/5.577/2.035 ms 2025-08-28 04:00:32,448 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-98134178 passed 2025-08-28 04:00:32,454 - INFO - Active control node from the Agent 10.0.0.65 is 10.0.0.41 2025-08-28 04:00:32,454 - INFO - Stoping the Control service in 10.0.0.41 2025-08-28 04:00:32,998 - INFO - 10.0.0.41 2025-08-28 04:00:32,998 - INFO - control:inactive 2025-08-28 04:00:32,998 - INFO - Contrail services control are down on nodes 10.0.0.41 2025-08-28 04:00:38,006 - INFO - Active control node from the Agent 10.0.0.65 is 10.0.0.19 2025-08-28 04:00:38,006 - INFO - Starting the Control service in 10.0.0.41 2025-08-28 04:00:38,319 - INFO - 10.0.0.41 2025-08-28 04:00:38,323 - INFO - control:initializing 2025-08-28 04:00:38,323 - DEBUG - defaultdict(, {'10.0.0.41': {'control': {'status': 'initializing', 'description': None}}}) 2025-08-28 04:00:38,324 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-08-28 04:00:43,410 - INFO - 10.0.0.41 2025-08-28 04:00:43,415 - INFO - control:active 2025-08-28 04:00:43,415 - INFO - Contrail services ['control'] are up on nodes 10.0.0.41 2025-08-28 04:00:48,536 - DEBUG - VM is in ACTIVE state now 2025-08-28 04:00:48,536 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:00:48,536 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5 2025-08-28 04:00:48,544 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/7116d704-0c73-45ec-a73e-0ed717b95a99 2025-08-28 04:00:48,553 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5 2025-08-28 04:00:48,560 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/7116d704-0c73-45ec-a73e-0ed717b95a99 2025-08-28 04:00:48,571 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/1c52c5b9-239a-45a0-b8f7-44240b2623ae 2025-08-28 04:00:48,581 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 04:00:48,581 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5 2025-08-28 04:00:48,588 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/7116d704-0c73-45ec-a73e-0ed717b95a99 2025-08-28 04:00:48,597 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/1c52c5b9-239a-45a0-b8f7-44240b2623ae 2025-08-28 04:00:48,605 - INFO - VM ctest-vn1_vm1-60404630 verfication in all API Servers passed 2025-08-28 04:00:48,622 - DEBUG - VM ctest-vn1_vm1-60404630 Tap interface: {'index': '3', 'name': 'tap7116d704-0c', 'uuid': '7116d704-0c73-45ec-a73e-0ed717b95a99', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296:ctest-vn1-29522296', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296', 'vm_uuid': 'badab9b2-b145-4df1-b61e-027fb2e759e5', 'vm_name': 'ctest-vn1_vm1-60404630', 'ip_addr': '192.168.1.3', 'mac_addr': '02:71:16:d7:04:0c', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '160', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-72692947:7116d704-0c73-45ec-a73e-0ed717b95a99', 'sg_uuid_list': ['9e2e8ff8-0b3f-400e-973b-a630f0dafd7f'], 'static_route_list': None, 'vm_project_uuid': '5631e639-3d56-4dbe-9639-2fd2c3fa5ac2', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-28 04:00:48,622 - DEBUG - Agent 10.0.0.65 vrf name: default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296:ctest-vn1-29522296 2025-08-28 04:00:48,626 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296:ctest-vn1-29522296', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296', 'table_label': '-1', 'vxlan_id': '12', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.193:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-08-28 04:00:48,703 - DEBUG - VM ctest-vn1_vm1-60404630 : Tap interface tap7116d704-0c is set to Active 2025-08-28 04:00:48,703 - DEBUG - Tap interface tap7116d704-0c detail : {'index': '3', 'name': 'tap7116d704-0c', 'uuid': '7116d704-0c73-45ec-a73e-0ed717b95a99', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296:ctest-vn1-29522296', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296', 'vm_uuid': 'badab9b2-b145-4df1-b61e-027fb2e759e5', 'vm_name': 'ctest-vn1_vm1-60404630', 'ip_addr': '192.168.1.3', 'mac_addr': '02:71:16:d7:04:0c', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '160', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-72692947:7116d704-0c73-45ec-a73e-0ed717b95a99', 'sg_uuid_list': ['9e2e8ff8-0b3f-400e-973b-a630f0dafd7f'], 'static_route_list': None, 'vm_project_uuid': '5631e639-3d56-4dbe-9639-2fd2c3fa5ac2', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-28 04:00:48,810 - DEBUG - Starting Layer 2 verification in Agent 2025-08-28 04:00:48,821 - DEBUG - Layer 2 path is seen for VM MAC 02:71:16:d7:04:0c in agent 10.0.0.65 2025-08-28 04:00:48,821 - DEBUG - Active layer 2 route in agent is present for VMI tap7116d704-0c 2025-08-28 04:00:48,822 - DEBUG - L2 label(29) matches bw route table and interface table 2025-08-28 04:00:48,827 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is found in Agent of node 10.0.0.65 2025-08-28 04:00:48,954 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is consistent in agent 10.0.0.65 2025-08-28 04:00:48,954 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.65 2025-08-28 04:00:48,954 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 verification for VM ctest-vn1_vm1-60404630 in Agent 10.0.0.65 passed 2025-08-28 04:00:49,061 - DEBUG - Starting all layer 2 verification in agent 10.0.0.65 2025-08-28 04:00:49,072 - DEBUG - Route for VM MAC 02:71:16:d7:04:0c is consistent in agent 10.0.0.65 2025-08-28 04:00:49,078 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is found in Agent of node 10.0.0.27 2025-08-28 04:00:49,204 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is consistent in agent 10.0.0.27 2025-08-28 04:00:49,204 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.27 2025-08-28 04:00:49,204 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 verification for VM ctest-vn1_vm1-60404630 in Agent 10.0.0.27 passed 2025-08-28 04:00:49,318 - DEBUG - Starting all layer 2 verification in agent 10.0.0.27 2025-08-28 04:00:49,327 - DEBUG - Route for VM MAC 02:71:16:d7:04:0c is consistent in agent 10.0.0.27 2025-08-28 04:00:49,328 - INFO - VM ctest-vn1_vm1-60404630 verifications in Compute nodes passed 2025-08-28 04:00:49,561 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.65 2025-08-28 04:00:49,566 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-08-28 04:00:49,663 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.27 2025-08-28 04:00:49,667 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.193, 24; Got 10.20.0.193,24 2025-08-28 04:00:49,668 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.27 2025-08-28 04:00:49,668 - INFO - Validated routes of VM ctest-vn1_vm1-60404630 in all vrouters 2025-08-28 04:00:50,160 - DEBUG - Starting all layer2 verification in 10.0.0.41 Control Node 2025-08-28 04:00:50,288 - DEBUG - Layer2 route found for VM MAC 02:71:16:d7:04:0c in Control-node 10.0.0.41 2025-08-28 04:00:50,288 - DEBUG - L2 Label for VM ctest-vn1_vm1-60404630 same between Control-node 10.0.0.41 and Agent, Expected: 29, Seen: 29 2025-08-28 04:00:50,289 - DEBUG - Starting all layer2 verification in 10.0.0.19 Control Node 2025-08-28 04:00:50,427 - DEBUG - Layer2 route found for VM MAC 02:71:16:d7:04:0c in Control-node 10.0.0.19 2025-08-28 04:00:50,427 - DEBUG - L2 Label for VM ctest-vn1_vm1-60404630 same between Control-node 10.0.0.19 and Agent, Expected: 29, Seen: 29 2025-08-28 04:00:50,427 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-08-28 04:00:50,548 - DEBUG - Layer2 route found for VM MAC 02:71:16:d7:04:0c in Control-node 10.0.0.38 2025-08-28 04:00:50,548 - DEBUG - L2 Label for VM ctest-vn1_vm1-60404630 same between Control-node 10.0.0.38 and Agent, Expected: 29, Seen: 29 2025-08-28 04:00:50,548 - INFO - Verification in Control-nodes for VM ctest-vn1_vm1-60404630 passed 2025-08-28 04:00:50,548 - DEBUG - Verifying the vm in opserver 2025-08-28 04:00:50,548 - DEBUG - Verifying in collector 10.0.0.41 ... 2025-08-28 04:00:50,548 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5?flat 2025-08-28 04:00:50,608 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-72692947:7116d704-0c73-45ec-a73e-0ed717b95a99?flat 2025-08-28 04:00:50,666 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-72692947:7116d704-0c73-45ec-a73e-0ed717b95a99?flat 2025-08-28 04:00:50,685 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5?flat 2025-08-28 04:00:50,699 - DEBUG - Verifying vm in vn uve 2025-08-28 04:00:50,699 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-72692947:7116d704-0c73-45ec-a73e-0ed717b95a99?flat 2025-08-28 04:00:50,759 - DEBUG - VM uve shows interface as {'vm_name': 'ctest-vn1_vm1-60404630', 'ip4_active': True, 'active': True, '__T': 1756353533537135, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296', 'mac_address': '02:71:16:d7:04:0c', 'label': 24, 'ip6_active': False, 'ip6_address': '::', 'gateway': '192.168.1.1', 'fixed_ip4_list': ['192.168.1.3'], 'is_health_check_active': True, 'tx_vlan': 65535, 'l2_active': True, 'port_mirror_enabled': False, 'vm_uuid': 'badab9b2-b145-4df1-b61e-027fb2e759e5', 'vn_uuid': '6c1038b4-167a-42ef-a098-3298511148db', 'ip_address': '192.168.1.3', 'admin_state': True, 'rx_vlan': 65535, 'vhostuser_mode': 0, 'uuid': '7116d704-0c73-45ec-a73e-0ed717b95a99'} 2025-08-28 04:00:50,759 - DEBUG - VM uve shows ip address as ['192.168.1.3', '::'] 2025-08-28 04:00:50,759 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 2025-08-28 04:00:50,759 - INFO - Verifying through opserver in 10.0.0.19 2025-08-28 04:00:50,759 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296?flat 2025-08-28 04:00:50,792 - DEBUG - expected vm list badab9b2-b145-4df1-b61e-027fb2e759e5 2025-08-28 04:00:50,792 - DEBUG - Extracted vm list ['badab9b2-b145-4df1-b61e-027fb2e759e5'] 2025-08-28 04:00:50,792 - INFO - VM badab9b2-b145-4df1-b61e-027fb2e759e5 is present in default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 2025-08-28 04:00:50,792 - DEBUG - Verifying vm in vrouter uve 2025-08-28 04:00:50,792 - DEBUG - Getting info from collector 10.0.0.41.. 2025-08-28 04:00:50,792 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5?flat 2025-08-28 04:00:50,851 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-3956-1'] 2025-08-28 04:00:50,851 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:00:50,903 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:00:50,936 - DEBUG - VM badab9b2-b145-4df1-b61e-027fb2e759e5 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:00:50,937 - DEBUG - Validated that VM ctest-vn1_vm1-60404630 is in Vrouter an-jenkins-deploy-platform-ansible-os-3956-1 UVE 2025-08-28 04:00:50,937 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-08-28 04:00:50,943 - DEBUG - Expected tap interface of VM uuid badab9b2-b145-4df1-b61e-027fb2e759e5 is default-domain:ctest-TestBasicVMVN0-72692947:7116d704-0c73-45ec-a73e-0ed717b95a99 2025-08-28 04:00:50,944 - DEBUG - Expected VN of VM uuid badab9b2-b145-4df1-b61e-027fb2e759e5 is default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 2025-08-28 04:00:50,944 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:00:50,994 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:00:51,024 - DEBUG - VM badab9b2-b145-4df1-b61e-027fb2e759e5 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:00:51,024 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-72692947:7116d704-0c73-45ec-a73e-0ed717b95a99 of vm badab9b2-b145-4df1-b61e-027fb2e759e5 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:00:51,024 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 of vm badab9b2-b145-4df1-b61e-027fb2e759e5 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:00:51,025 - DEBUG - Verifying the badab9b2-b145-4df1-b61e-027fb2e759e5 virtual network link through opserver 10.0.0.19 2025-08-28 04:00:51,025 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-08-28 04:00:51,038 - DEBUG - VM link and name as {'name': 'badab9b2-b145-4df1-b61e-027fb2e759e5', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5?flat'} 2025-08-28 04:00:51,038 - DEBUG - Verifying the badab9b2-b145-4df1-b61e-027fb2e759e5 virtual network link through opserver 10.0.0.38 2025-08-28 04:00:51,038 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-28 04:00:51,055 - DEBUG - VM link and name as {'name': 'badab9b2-b145-4df1-b61e-027fb2e759e5', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5?flat'} 2025-08-28 04:00:51,055 - DEBUG - Verifying the badab9b2-b145-4df1-b61e-027fb2e759e5 virtual network link through opserver 10.0.0.41 2025-08-28 04:00:51,056 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines 2025-08-28 04:00:51,068 - DEBUG - VM link and name as {'name': 'badab9b2-b145-4df1-b61e-027fb2e759e5', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5?flat'} 2025-08-28 04:00:51,068 - INFO - VM ctest-vn1_vm1-60404630 validations in Opserver passed 2025-08-28 04:00:51,150 - DEBUG - VM is in ACTIVE state now 2025-08-28 04:00:51,150 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:00:51,150 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de59cee3-9395-4745-bd10-ef98d546c004 2025-08-28 04:00:51,159 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/57101bdd-2a06-416f-9478-a6341fe66b2c 2025-08-28 04:00:51,169 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de59cee3-9395-4745-bd10-ef98d546c004 2025-08-28 04:00:51,175 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/57101bdd-2a06-416f-9478-a6341fe66b2c 2025-08-28 04:00:51,185 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/7468379f-b414-4e61-8e4f-28a1db68c221 2025-08-28 04:00:51,193 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 04:00:51,194 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de59cee3-9395-4745-bd10-ef98d546c004 2025-08-28 04:00:51,200 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/57101bdd-2a06-416f-9478-a6341fe66b2c 2025-08-28 04:00:51,209 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/7468379f-b414-4e61-8e4f-28a1db68c221 2025-08-28 04:00:51,217 - INFO - VM ctest-vn1_vm2-98134178 verfication in all API Servers passed 2025-08-28 04:00:51,235 - DEBUG - VM ctest-vn1_vm2-98134178 Tap interface: {'index': '3', 'name': 'tap57101bdd-2a', 'uuid': '57101bdd-2a06-416f-9478-a6341fe66b2c', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296:ctest-vn1-29522296', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296', 'vm_uuid': 'de59cee3-9395-4745-bd10-ef98d546c004', 'vm_name': 'ctest-vn1_vm2-98134178', 'ip_addr': '192.168.1.4', 'mac_addr': '02:57:10:1b:dd:2a', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-72692947:57101bdd-2a06-416f-9478-a6341fe66b2c', 'sg_uuid_list': ['9e2e8ff8-0b3f-400e-973b-a630f0dafd7f'], 'static_route_list': None, 'vm_project_uuid': '5631e639-3d56-4dbe-9639-2fd2c3fa5ac2', 'admin_state': 'Enabled', 'flow_key_idx': '29', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-28 04:00:51,236 - DEBUG - Agent 10.0.0.27 vrf name: default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296:ctest-vn1-29522296 2025-08-28 04:00:51,240 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296:ctest-vn1-29522296', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296', 'table_label': '-1', 'vxlan_id': '12', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.192:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-08-28 04:00:51,323 - DEBUG - VM ctest-vn1_vm2-98134178 : Tap interface tap57101bdd-2a is set to Active 2025-08-28 04:00:51,323 - DEBUG - Tap interface tap57101bdd-2a detail : {'index': '3', 'name': 'tap57101bdd-2a', 'uuid': '57101bdd-2a06-416f-9478-a6341fe66b2c', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296:ctest-vn1-29522296', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296', 'vm_uuid': 'de59cee3-9395-4745-bd10-ef98d546c004', 'vm_name': 'ctest-vn1_vm2-98134178', 'ip_addr': '192.168.1.4', 'mac_addr': '02:57:10:1b:dd:2a', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-72692947:57101bdd-2a06-416f-9478-a6341fe66b2c', 'sg_uuid_list': ['9e2e8ff8-0b3f-400e-973b-a630f0dafd7f'], 'static_route_list': None, 'vm_project_uuid': '5631e639-3d56-4dbe-9639-2fd2c3fa5ac2', 'admin_state': 'Enabled', 'flow_key_idx': '29', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-28 04:00:51,429 - DEBUG - Starting Layer 2 verification in Agent 2025-08-28 04:00:51,441 - DEBUG - Layer 2 path is seen for VM MAC 02:57:10:1b:dd:2a in agent 10.0.0.27 2025-08-28 04:00:51,441 - DEBUG - Active layer 2 route in agent is present for VMI tap57101bdd-2a 2025-08-28 04:00:51,441 - DEBUG - L2 label(29) matches bw route table and interface table 2025-08-28 04:00:51,446 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is found in Agent of node 10.0.0.65 2025-08-28 04:00:51,573 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is consistent in agent 10.0.0.65 2025-08-28 04:00:51,573 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.65 2025-08-28 04:00:51,573 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 verification for VM ctest-vn1_vm2-98134178 in Agent 10.0.0.65 passed 2025-08-28 04:00:51,690 - DEBUG - Starting all layer 2 verification in agent 10.0.0.65 2025-08-28 04:00:51,700 - DEBUG - Route for VM MAC 02:57:10:1b:dd:2a is consistent in agent 10.0.0.65 2025-08-28 04:00:51,706 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is found in Agent of node 10.0.0.27 2025-08-28 04:00:51,834 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 is consistent in agent 10.0.0.27 2025-08-28 04:00:51,835 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.27 2025-08-28 04:00:51,835 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 verification for VM ctest-vn1_vm2-98134178 in Agent 10.0.0.27 passed 2025-08-28 04:00:51,946 - DEBUG - Starting all layer 2 verification in agent 10.0.0.27 2025-08-28 04:00:51,957 - DEBUG - Route for VM MAC 02:57:10:1b:dd:2a is consistent in agent 10.0.0.27 2025-08-28 04:00:51,957 - INFO - VM ctest-vn1_vm2-98134178 verifications in Compute nodes passed 2025-08-28 04:00:52,214 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.65 2025-08-28 04:00:52,219 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.192, 24; Got 10.20.0.192,24 2025-08-28 04:00:52,219 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.65 2025-08-28 04:00:52,330 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.27 2025-08-28 04:00:52,335 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-08-28 04:00:52,335 - INFO - Validated routes of VM ctest-vn1_vm2-98134178 in all vrouters 2025-08-28 04:00:52,829 - DEBUG - Starting all layer2 verification in 10.0.0.41 Control Node 2025-08-28 04:00:52,955 - DEBUG - Layer2 route found for VM MAC 02:57:10:1b:dd:2a in Control-node 10.0.0.41 2025-08-28 04:00:52,955 - DEBUG - L2 Label for VM ctest-vn1_vm2-98134178 same between Control-node 10.0.0.41 and Agent, Expected: 29, Seen: 29 2025-08-28 04:00:52,955 - DEBUG - Starting all layer2 verification in 10.0.0.19 Control Node 2025-08-28 04:00:53,076 - DEBUG - Layer2 route found for VM MAC 02:57:10:1b:dd:2a in Control-node 10.0.0.19 2025-08-28 04:00:53,077 - DEBUG - L2 Label for VM ctest-vn1_vm2-98134178 same between Control-node 10.0.0.19 and Agent, Expected: 29, Seen: 29 2025-08-28 04:00:53,077 - DEBUG - Starting all layer2 verification in 10.0.0.38 Control Node 2025-08-28 04:00:53,204 - DEBUG - Layer2 route found for VM MAC 02:57:10:1b:dd:2a in Control-node 10.0.0.38 2025-08-28 04:00:53,204 - DEBUG - L2 Label for VM ctest-vn1_vm2-98134178 same between Control-node 10.0.0.38 and Agent, Expected: 29, Seen: 29 2025-08-28 04:00:53,204 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-98134178 passed 2025-08-28 04:00:53,204 - DEBUG - Verifying the vm in opserver 2025-08-28 04:00:53,204 - DEBUG - Verifying in collector 10.0.0.41 ... 2025-08-28 04:00:53,205 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/de59cee3-9395-4745-bd10-ef98d546c004?flat 2025-08-28 04:00:53,219 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-72692947:57101bdd-2a06-416f-9478-a6341fe66b2c?flat 2025-08-28 04:00:53,236 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-72692947:57101bdd-2a06-416f-9478-a6341fe66b2c?flat 2025-08-28 04:00:53,253 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/de59cee3-9395-4745-bd10-ef98d546c004?flat 2025-08-28 04:00:53,268 - DEBUG - Verifying vm in vn uve 2025-08-28 04:00:53,268 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-72692947:57101bdd-2a06-416f-9478-a6341fe66b2c?flat 2025-08-28 04:00:53,287 - DEBUG - VM uve shows interface as {'fixed_ip4_list': ['192.168.1.4'], 'active': True, 'uuid': '57101bdd-2a06-416f-9478-a6341fe66b2c', 'is_health_check_active': True, 'port_mirror_enabled': False, 'l2_active': True, 'tx_vlan': 65535, 'ip4_active': True, 'vn_uuid': '6c1038b4-167a-42ef-a098-3298511148db', 'label': 24, 'vm_name': 'ctest-vn1_vm2-98134178', 'vm_uuid': 'de59cee3-9395-4745-bd10-ef98d546c004', 'mac_address': '02:57:10:1b:dd:2a', 'rx_vlan': 65535, 'gateway': '192.168.1.1', 'ip_address': '192.168.1.4', 'admin_state': True, 'ip6_active': False, '__T': 1756353542409933, 'vhostuser_mode': 0, 'ip6_address': '::', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296'} 2025-08-28 04:00:53,287 - DEBUG - VM uve shows ip address as ['192.168.1.4', '::'] 2025-08-28 04:00:53,287 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 2025-08-28 04:00:53,287 - INFO - Verifying through opserver in 10.0.0.19 2025-08-28 04:00:53,287 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296?flat 2025-08-28 04:00:53,311 - DEBUG - expected vm list de59cee3-9395-4745-bd10-ef98d546c004 2025-08-28 04:00:53,311 - DEBUG - Extracted vm list ['de59cee3-9395-4745-bd10-ef98d546c004'] 2025-08-28 04:00:53,311 - INFO - VM de59cee3-9395-4745-bd10-ef98d546c004 is present in default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 2025-08-28 04:00:53,311 - DEBUG - Verifying vm in vrouter uve 2025-08-28 04:00:53,311 - DEBUG - Getting info from collector 10.0.0.41.. 2025-08-28 04:00:53,311 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/de59cee3-9395-4745-bd10-ef98d546c004?flat 2025-08-28 04:00:53,326 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-3956-2'] 2025-08-28 04:00:53,326 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-2:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:00:53,374 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-2?flat 2025-08-28 04:00:53,451 - DEBUG - VM de59cee3-9395-4745-bd10-ef98d546c004 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-2 uve 2025-08-28 04:00:53,451 - DEBUG - Validated that VM ctest-vn1_vm2-98134178 is in Vrouter an-jenkins-deploy-platform-ansible-os-3956-2 UVE 2025-08-28 04:00:53,451 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-08-28 04:00:53,458 - DEBUG - Expected tap interface of VM uuid de59cee3-9395-4745-bd10-ef98d546c004 is default-domain:ctest-TestBasicVMVN0-72692947:57101bdd-2a06-416f-9478-a6341fe66b2c 2025-08-28 04:00:53,458 - DEBUG - Expected VN of VM uuid de59cee3-9395-4745-bd10-ef98d546c004 is default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 2025-08-28 04:00:53,458 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-2:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:00:53,505 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-2?flat 2025-08-28 04:00:53,611 - DEBUG - VM de59cee3-9395-4745-bd10-ef98d546c004 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-2 uve 2025-08-28 04:00:53,611 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-72692947:57101bdd-2a06-416f-9478-a6341fe66b2c of vm de59cee3-9395-4745-bd10-ef98d546c004 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-2 uve 2025-08-28 04:00:53,611 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 of vm de59cee3-9395-4745-bd10-ef98d546c004 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-2 uve 2025-08-28 04:00:53,611 - DEBUG - Verifying the de59cee3-9395-4745-bd10-ef98d546c004 virtual network link through opserver 10.0.0.19 2025-08-28 04:00:53,611 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-08-28 04:00:53,622 - DEBUG - VM link and name as {'name': 'de59cee3-9395-4745-bd10-ef98d546c004', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-machine/de59cee3-9395-4745-bd10-ef98d546c004?flat'} 2025-08-28 04:00:53,622 - DEBUG - Verifying the de59cee3-9395-4745-bd10-ef98d546c004 virtual network link through opserver 10.0.0.38 2025-08-28 04:00:53,622 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-28 04:00:53,634 - DEBUG - VM link and name as {'name': 'de59cee3-9395-4745-bd10-ef98d546c004', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/de59cee3-9395-4745-bd10-ef98d546c004?flat'} 2025-08-28 04:00:53,634 - DEBUG - Verifying the de59cee3-9395-4745-bd10-ef98d546c004 virtual network link through opserver 10.0.0.41 2025-08-28 04:00:53,634 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines 2025-08-28 04:00:53,646 - DEBUG - VM link and name as {'name': 'de59cee3-9395-4745-bd10-ef98d546c004', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-machine/de59cee3-9395-4745-bd10-ef98d546c004?flat'} 2025-08-28 04:00:53,647 - INFO - VM ctest-vn1_vm2-98134178 validations in Opserver passed 2025-08-28 04:00:53,647 - INFO - Checking the ping between the VM again 2025-08-28 04:00:53,647 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:00:53,647 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.65, gateway password: c0ntrail123 2025-08-28 04:00:53,647 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4 2025-08-28 04:00:56,088 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data. 64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=3.77 ms 64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.07 ms 64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=0.941 ms --- 192.168.1.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 0.941/1.928/3.775/1.307 ms 2025-08-28 04:00:56,088 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-60404630 passed 2025-08-28 04:00:56,088 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:00:56,088 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123 2025-08-28 04:00:56,088 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3 2025-08-28 04:00:58,505 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. 64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=4.41 ms 64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.12 ms 64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.04 ms --- 192.168.1.3 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.040/2.194/4.415/1.571 ms 2025-08-28 04:00:58,506 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-98134178 passed 2025-08-28 04:00:58,506 - INFO - Deleting VM ctest-vn1_vm2-98134178 2025-08-28 04:00:58,575 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 04:00:58,575 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de59cee3-9395-4745-bd10-ef98d546c004 2025-08-28 04:00:58,586 - DEBUG - VM ID de59cee3-9395-4745-bd10-ef98d546c004 of VM ctest-vn1_vm2-98134178 is still found in API Server 2025-08-28 04:01:00,587 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 04:01:00,587 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de59cee3-9395-4745-bd10-ef98d546c004 2025-08-28 04:01:00,594 - DEBUG - Response Code: 404 2025-08-28 04:01:00,594 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de59cee3-9395-4745-bd10-ef98d546c004 2025-08-28 04:01:00,600 - DEBUG - Response Code: 404 2025-08-28 04:01:00,601 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de59cee3-9395-4745-bd10-ef98d546c004 2025-08-28 04:01:00,607 - DEBUG - Response Code: 404 2025-08-28 04:01:00,607 - INFO - VM ctest-vn1_vm2-98134178 is fully removed in API-Server 2025-08-28 04:01:00,756 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-08-28 04:01:00,909 - DEBUG - VM route 192.168.1.4 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 of compute 10.0.0.65 2025-08-28 04:01:00,914 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 2025-08-28 04:01:00,914 - DEBUG - Validated that vrouter 10.0.0.27 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 2025-08-28 04:01:00,914 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 2025-08-28 04:01:00,914 - INFO - VM ctest-vn1_vm2-98134178 is removed in Compute, and routes are removed in all compute nodes 2025-08-28 04:01:01,161 - INFO - Routes for VM ctest-vn1_vm2-98134178 is removed in all control-nodes 2025-08-28 04:01:01,161 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-2:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:01:01,218 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-2?flat 2025-08-28 04:01:01,257 - DEBUG - VM de59cee3-9395-4745-bd10-ef98d546c004 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-2 uve 2025-08-28 04:01:01,257 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:01:01,258 - DEBUG - interface for vm de59cee3-9395-4745-bd10-ef98d546c004 deleted from vrouter uve ... 2025-08-28 04:01:01,258 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:01:04,258 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-2:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:01:04,309 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-2?flat 2025-08-28 04:01:04,341 - DEBUG - VM de59cee3-9395-4745-bd10-ef98d546c004 is not present in vrouter an-jenkins-deploy-platform-ansible-os-3956-2 uve 2025-08-28 04:01:04,342 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:01:04,342 - DEBUG - interface for vm de59cee3-9395-4745-bd10-ef98d546c004 deleted from vrouter uve ... 2025-08-28 04:01:04,342 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:01:04,342 - DEBUG - Validated that VM de59cee3-9395-4745-bd10-ef98d546c004 is removed in Vrouter UVE 2025-08-28 04:01:04,342 - DEBUG - Verifying through opserver in 10.0.0.19 2025-08-28 04:01:04,342 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296?flat 2025-08-28 04:01:04,376 - DEBUG - de59cee3-9395-4745-bd10-ef98d546c004 BM not in default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 uve 2025-08-28 04:01:04,376 - DEBUG - Validated that VM de59cee3-9395-4745-bd10-ef98d546c004 is not present in the VN UVE 2025-08-28 04:01:04,376 - DEBUG - Verifying the de59cee3-9395-4745-bd10-ef98d546c004 virtual network link through opserver 10.0.0.19 2025-08-28 04:01:04,376 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-08-28 04:01:04,389 - INFO - de59cee3-9395-4745-bd10-ef98d546c004 vm uve deleted from opserver 2025-08-28 04:01:04,389 - DEBUG - Verifying the de59cee3-9395-4745-bd10-ef98d546c004 virtual network link through opserver 10.0.0.38 2025-08-28 04:01:04,389 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-28 04:01:04,403 - INFO - de59cee3-9395-4745-bd10-ef98d546c004 vm uve deleted from opserver 2025-08-28 04:01:04,403 - DEBUG - Verifying the de59cee3-9395-4745-bd10-ef98d546c004 virtual network link through opserver 10.0.0.41 2025-08-28 04:01:04,403 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines 2025-08-28 04:01:04,416 - INFO - de59cee3-9395-4745-bd10-ef98d546c004 vm uve deleted from opserver 2025-08-28 04:01:04,416 - INFO - Validated that VM de59cee3-9395-4745-bd10-ef98d546c004 is removed from Opserver 2025-08-28 04:01:04,416 - INFO - Deleting VM ctest-vn1_vm1-60404630 2025-08-28 04:01:04,485 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 04:01:04,485 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5 2025-08-28 04:01:04,493 - DEBUG - VM ID badab9b2-b145-4df1-b61e-027fb2e759e5 of VM ctest-vn1_vm1-60404630 is still found in API Server 2025-08-28 04:01:06,494 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 04:01:06,494 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5 2025-08-28 04:01:06,501 - DEBUG - Response Code: 404 2025-08-28 04:01:06,501 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5 2025-08-28 04:01:06,506 - DEBUG - Response Code: 404 2025-08-28 04:01:06,506 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/badab9b2-b145-4df1-b61e-027fb2e759e5 2025-08-28 04:01:06,512 - DEBUG - Response Code: 404 2025-08-28 04:01:06,512 - INFO - VM ctest-vn1_vm1-60404630 is fully removed in API-Server 2025-08-28 04:01:06,664 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-08-28 04:01:06,702 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 2025-08-28 04:01:06,713 - DEBUG - Validated that vrouter 10.0.0.27 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 2025-08-28 04:01:06,713 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 2025-08-28 04:01:06,713 - INFO - VM ctest-vn1_vm1-60404630 is removed in Compute, and routes are removed in all compute nodes 2025-08-28 04:01:06,954 - INFO - Routes for VM ctest-vn1_vm1-60404630 is removed in all control-nodes 2025-08-28 04:01:06,954 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:01:07,001 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:01:07,032 - DEBUG - VM badab9b2-b145-4df1-b61e-027fb2e759e5 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:01:07,032 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:01:07,032 - DEBUG - interface for vm badab9b2-b145-4df1-b61e-027fb2e759e5 deleted from vrouter uve ... 2025-08-28 04:01:07,033 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:01:10,033 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:01:10,078 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:01:10,108 - DEBUG - VM badab9b2-b145-4df1-b61e-027fb2e759e5 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:01:10,108 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:01:10,108 - DEBUG - interface for vm badab9b2-b145-4df1-b61e-027fb2e759e5 deleted from vrouter uve ... 2025-08-28 04:01:10,108 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:01:13,109 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:01:13,155 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:01:13,184 - DEBUG - VM badab9b2-b145-4df1-b61e-027fb2e759e5 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:01:13,184 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:01:13,184 - DEBUG - interface for vm badab9b2-b145-4df1-b61e-027fb2e759e5 deleted from vrouter uve ... 2025-08-28 04:01:13,184 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:01:16,184 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:01:16,235 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:01:16,265 - DEBUG - VM badab9b2-b145-4df1-b61e-027fb2e759e5 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:01:16,265 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:01:16,265 - DEBUG - interface for vm badab9b2-b145-4df1-b61e-027fb2e759e5 deleted from vrouter uve ... 2025-08-28 04:01:16,265 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:01:19,266 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:01:19,325 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:01:19,364 - DEBUG - VM badab9b2-b145-4df1-b61e-027fb2e759e5 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:01:19,364 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:01:19,364 - DEBUG - interface for vm badab9b2-b145-4df1-b61e-027fb2e759e5 deleted from vrouter uve ... 2025-08-28 04:01:19,364 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:01:22,365 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:01:22,412 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:01:22,440 - DEBUG - VM badab9b2-b145-4df1-b61e-027fb2e759e5 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:01:22,440 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:01:22,440 - DEBUG - interface for vm badab9b2-b145-4df1-b61e-027fb2e759e5 deleted from vrouter uve ... 2025-08-28 04:01:22,440 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:01:25,441 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:01:25,489 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:01:25,519 - DEBUG - VM badab9b2-b145-4df1-b61e-027fb2e759e5 is not present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:01:25,519 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:01:25,519 - DEBUG - interface for vm badab9b2-b145-4df1-b61e-027fb2e759e5 deleted from vrouter uve ... 2025-08-28 04:01:25,519 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:01:25,519 - DEBUG - Validated that VM badab9b2-b145-4df1-b61e-027fb2e759e5 is removed in Vrouter UVE 2025-08-28 04:01:25,519 - DEBUG - Verifying through opserver in 10.0.0.19 2025-08-28 04:01:25,519 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296?flat 2025-08-28 04:01:25,544 - DEBUG - badab9b2-b145-4df1-b61e-027fb2e759e5 BM not in default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-29522296 uve 2025-08-28 04:01:25,544 - DEBUG - Validated that VM badab9b2-b145-4df1-b61e-027fb2e759e5 is not present in the VN UVE 2025-08-28 04:01:25,544 - DEBUG - Verifying the badab9b2-b145-4df1-b61e-027fb2e759e5 virtual network link through opserver 10.0.0.19 2025-08-28 04:01:25,544 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-08-28 04:01:25,556 - INFO - badab9b2-b145-4df1-b61e-027fb2e759e5 vm uve deleted from opserver 2025-08-28 04:01:25,556 - DEBUG - Verifying the badab9b2-b145-4df1-b61e-027fb2e759e5 virtual network link through opserver 10.0.0.38 2025-08-28 04:01:25,556 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-28 04:01:25,569 - INFO - badab9b2-b145-4df1-b61e-027fb2e759e5 vm uve deleted from opserver 2025-08-28 04:01:25,569 - DEBUG - Verifying the badab9b2-b145-4df1-b61e-027fb2e759e5 virtual network link through opserver 10.0.0.41 2025-08-28 04:01:25,569 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines 2025-08-28 04:01:25,580 - INFO - badab9b2-b145-4df1-b61e-027fb2e759e5 vm uve deleted from opserver 2025-08-28 04:01:25,580 - INFO - Validated that VM badab9b2-b145-4df1-b61e-027fb2e759e5 is removed from Opserver 2025-08-28 04:01:25,580 - INFO - Deleting VN ctest-vn1-29522296 2025-08-28 04:01:25,755 - DEBUG - Response for deleting network () 2025-08-28 04:01:25,755 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/4ee685b5-7946-48da-b6df-484d200631ec 2025-08-28 04:01:25,761 - DEBUG - Response Code: 404 2025-08-28 04:01:25,761 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 04:01:25,766 - DEBUG - Requesting: http://10.0.0.19:8082/domain/692c5a68-a199-4232-a07c-1cd4918de9d4 2025-08-28 04:01:25,804 - DEBUG - Requesting: http://10.0.0.19:8082/project/5631e639-3d56-4dbe-9639-2fd2c3fa5ac2 2025-08-28 04:01:25,904 - INFO - Validated that VN ctest-vn1-29522296 is not found in API Server 2025-08-28 04:01:25,914 - DEBUG - VN ctest-vn1-29522296 is not present in Agent 10.0.0.65 2025-08-28 04:01:25,922 - DEBUG - VN ctest-vn1-29522296 is not present in Agent 10.0.0.27 2025-08-28 04:01:25,922 - INFO - Validated that VN ctest-vn1-29522296 is not in any agent 2025-08-28 04:01:25,949 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-29522296 info 2025-08-28 04:01:27,213 - DEBUG - Skipping xmpp flap check 2025-08-28 04:01:27,214 - INFO - END TEST : test_control_node_switchover : PASSED[0:02:54] 2025-08-28 04:01:27,214 - INFO - -------------------------------------------------------------------------------- 2025-08-28 04:01:27,216 - INFO - ================================================================================ 2025-08-28 04:01:27,216 - INFO - STARTING TEST : test_process_restart_in_policy_between_vns 2025-08-28 04:01:27,216 - INFO - TEST DESCRIPTION : Test to validate that with policy having rule to check icmp fwding between VMs on different VNs , ping between VMs should pass with process restarts 1. Pick 2 VN's from resource pool which has one VM each 2. Create policy with icmp allow rule between those VN's and bind it networks 3. Ping from one VM to another VM 4. Restart process 'vrouter' and 'control' on setup 5. Ping again between VM's after process restart Pass criteria: Step 2,3,4 and 5 should pass 2025-08-28 04:01:28,482 - DEBUG - Skipping xmpp flap check 2025-08-28 04:01:28,482 - INFO - Initial checks done. Running the testcase now 2025-08-28 04:01:28,482 - INFO - 2025-08-28 04:01:28,488 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-08-28 04:01:28,761 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-08-28 04:01:29,305 - INFO - Created VN ctest-vn1-50579577, UUID :b897aa08-efeb-466e-89f1-72b7f9200eae 2025-08-28 04:01:29,614 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 04:01:29,619 - DEBUG - Requesting: http://10.0.0.19:8082/domain/692c5a68-a199-4232-a07c-1cd4918de9d4 2025-08-28 04:01:29,630 - DEBUG - Requesting: http://10.0.0.19:8082/project/5631e639-3d56-4dbe-9639-2fd2c3fa5ac2 2025-08-28 04:01:29,723 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/b897aa08-efeb-466e-89f1-72b7f9200eae 2025-08-28 04:01:29,735 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/b897aa08-efeb-466e-89f1-72b7f9200eae 2025-08-28 04:01:29,744 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/36da7fbc-411a-4bba-ada9-874f8db9143c 2025-08-28 04:01:29,753 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/36da7fbc-411a-4bba-ada9-874f8db9143c 2025-08-28 04:01:29,761 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/80ee3fea-82cb-4a45-8e77-aa1b8e938f4b 2025-08-28 04:01:29,768 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 04:01:29,768 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/b897aa08-efeb-466e-89f1-72b7f9200eae 2025-08-28 04:01:29,778 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/36da7fbc-411a-4bba-ada9-874f8db9143c 2025-08-28 04:01:29,792 - INFO - Verified VN network id 12 for VN b897aa08-efeb-466e-89f1-72b7f9200eae 2025-08-28 04:01:29,792 - INFO - Verifications in API Server for VN ctest-vn1-50579577 passed 2025-08-28 04:01:29,792 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/b897aa08-efeb-466e-89f1-72b7f9200eae 2025-08-28 04:01:29,802 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/36da7fbc-411a-4bba-ada9-874f8db9143c 2025-08-28 04:01:29,809 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/36da7fbc-411a-4bba-ada9-874f8db9143c 2025-08-28 04:01:29,816 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/80ee3fea-82cb-4a45-8e77-aa1b8e938f4b 2025-08-28 04:01:29,827 - DEBUG - Control-node 10.0.0.19 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-72692947', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13301286979823683182', 'uuid-lslong': '9939851987067604654'}, 'enable': 'true', 'created': '2025-08-28T04:01:29', 'last-modified': '2025-08-28T04:01:29', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.392101'} 2025-08-28 04:01:29,832 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 04:01:29,840 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-72692947', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13301286979823683182', 'uuid-lslong': '9939851987067604654'}, 'enable': 'true', 'created': '2025-08-28T04:01:29', 'last-modified': '2025-08-28T04:01:29', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.404344'} 2025-08-28 04:01:29,846 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 04:01:29,851 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-72692947', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13301286979823683182', 'uuid-lslong': '9939851987067604654'}, 'enable': 'true', 'created': '2025-08-28T04:01:29', 'last-modified': '2025-08-28T04:01:29', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.416516'} 2025-08-28 04:01:29,857 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 04:01:29,858 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-50579577 passed 2025-08-28 04:01:29,858 - DEBUG - ====Verifying policy data for ctest-vn1-50579577 in API_Server ====== 2025-08-28 04:01:29,858 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 04:01:29,864 - DEBUG - Requesting: http://10.0.0.19:8082/domain/692c5a68-a199-4232-a07c-1cd4918de9d4 2025-08-28 04:01:29,875 - DEBUG - Requesting: http://10.0.0.19:8082/project/5631e639-3d56-4dbe-9639-2fd2c3fa5ac2 2025-08-28 04:01:29,973 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/b897aa08-efeb-466e-89f1-72b7f9200eae 2025-08-28 04:01:29,984 - DEBUG - =>VN ctest-vn1-50579577 has no policy to be verified 2025-08-28 04:01:29,984 - DEBUG - Verifying the vn in opserver 2025-08-28 04:01:29,984 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 virtual network link through opserver 10.0.0.19 2025-08-28 04:01:29,985 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2025-08-28 04:01:30,000 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577?flat'} 2025-08-28 04:01:30,000 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 virtual network link through opserver 10.0.0.38 2025-08-28 04:01:30,000 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-28 04:01:30,014 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577?flat'} 2025-08-28 04:01:30,014 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 virtual network link through opserver 10.0.0.41 2025-08-28 04:01:30,014 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-08-28 04:01:30,025 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577?flat'} 2025-08-28 04:01:30,025 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is found in opserver 2025-08-28 04:01:30,044 - DEBUG - Do not have enough data to verify VN in agent 2025-08-28 04:01:30,052 - DEBUG - VRF ids for VN ctest-vn1-50579577: {} 2025-08-28 04:01:30,726 - INFO - Created VN ctest-vn2-66914093, UUID :c7e4e6f2-739e-41ec-9ed8-5516e9bfc536 2025-08-28 04:01:31,058 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 04:01:31,065 - DEBUG - Requesting: http://10.0.0.19:8082/domain/692c5a68-a199-4232-a07c-1cd4918de9d4 2025-08-28 04:01:31,075 - DEBUG - Requesting: http://10.0.0.19:8082/project/5631e639-3d56-4dbe-9639-2fd2c3fa5ac2 2025-08-28 04:01:31,172 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/c7e4e6f2-739e-41ec-9ed8-5516e9bfc536 2025-08-28 04:01:31,185 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/c7e4e6f2-739e-41ec-9ed8-5516e9bfc536 2025-08-28 04:01:31,194 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/7fb41dd1-1da1-4362-9085-853e726280ec 2025-08-28 04:01:31,203 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/7fb41dd1-1da1-4362-9085-853e726280ec 2025-08-28 04:01:31,210 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/7208903e-cae1-4622-bde5-4bc547ffda7e 2025-08-28 04:01:31,217 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-28 04:01:31,217 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/c7e4e6f2-739e-41ec-9ed8-5516e9bfc536 2025-08-28 04:01:31,227 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/7fb41dd1-1da1-4362-9085-853e726280ec 2025-08-28 04:01:31,242 - INFO - Verified VN network id 13 for VN c7e4e6f2-739e-41ec-9ed8-5516e9bfc536 2025-08-28 04:01:31,242 - INFO - Verifications in API Server for VN ctest-vn2-66914093 passed 2025-08-28 04:01:31,242 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/c7e4e6f2-739e-41ec-9ed8-5516e9bfc536 2025-08-28 04:01:31,253 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/7fb41dd1-1da1-4362-9085-853e726280ec 2025-08-28 04:01:31,262 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/7fb41dd1-1da1-4362-9085-853e726280ec 2025-08-28 04:01:31,269 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/7208903e-cae1-4622-bde5-4bc547ffda7e 2025-08-28 04:01:31,280 - DEBUG - Control-node 10.0.0.19 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-72692947', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14403891437233914348', 'uuid-lslong': '11445992009861416246'}, 'enable': 'true', 'created': '2025-08-28T04:01:30', 'last-modified': '2025-08-28T04:01:30', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.397680'} 2025-08-28 04:01:31,285 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-28 04:01:31,292 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-72692947', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14403891437233914348', 'uuid-lslong': '11445992009861416246'}, 'enable': 'true', 'created': '2025-08-28T04:01:30', 'last-modified': '2025-08-28T04:01:30', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.409457'} 2025-08-28 04:01:31,303 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-28 04:01:31,312 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-72692947', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14403891437233914348', 'uuid-lslong': '11445992009861416246'}, 'enable': 'true', 'created': '2025-08-28T04:01:30', 'last-modified': '2025-08-28T04:01:30', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.430179'} 2025-08-28 04:01:31,320 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-28 04:01:31,320 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-66914093 passed 2025-08-28 04:01:31,320 - DEBUG - ====Verifying policy data for ctest-vn2-66914093 in API_Server ====== 2025-08-28 04:01:31,320 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 04:01:31,330 - DEBUG - Requesting: http://10.0.0.19:8082/domain/692c5a68-a199-4232-a07c-1cd4918de9d4 2025-08-28 04:01:31,344 - DEBUG - Requesting: http://10.0.0.19:8082/project/5631e639-3d56-4dbe-9639-2fd2c3fa5ac2 2025-08-28 04:01:31,459 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/c7e4e6f2-739e-41ec-9ed8-5516e9bfc536 2025-08-28 04:01:31,470 - DEBUG - =>VN ctest-vn2-66914093 has no policy to be verified 2025-08-28 04:01:31,470 - DEBUG - Verifying the vn in opserver 2025-08-28 04:01:31,470 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 virtual network link through opserver 10.0.0.19 2025-08-28 04:01:31,470 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2025-08-28 04:01:31,484 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093?flat'} 2025-08-28 04:01:31,484 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 virtual network link through opserver 10.0.0.38 2025-08-28 04:01:31,484 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-28 04:01:31,497 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093?flat'} 2025-08-28 04:01:31,497 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 virtual network link through opserver 10.0.0.41 2025-08-28 04:01:31,497 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-08-28 04:01:31,510 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093?flat'} 2025-08-28 04:01:31,510 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 is found in opserver 2025-08-28 04:01:31,528 - DEBUG - Do not have enough data to verify VN in agent 2025-08-28 04:01:31,537 - DEBUG - VRF ids for VN ctest-vn2-66914093: {} 2025-08-28 04:01:33,047 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3956-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3956-1) 2025-08-28 04:01:34,467 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3956-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3956-2) 2025-08-28 04:01:34,467 - INFO - Waiting for VM ctest-vn1_vm1-24291330 to be up.. 2025-08-28 04:01:34,554 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-08-28 04:01:39,632 - DEBUG - VM is in ACTIVE state now 2025-08-28 04:01:39,632 - INFO - VM name : ctest-vn1_vm1-24291330 2025-08-28 04:01:39,703 - DEBUG - VM ctest-vn1_vm1-24291330 ID is 671f96fc-5f83-440a-a0d9-515a24d69de6 2025-08-28 04:01:39,703 - DEBUG - VM ctest-vn1_vm1-24291330 launched on Node an-jenkins-deploy-platform-ansible-os-3956-1 2025-08-28 04:01:39,781 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/671f96fc-5f83-440a-a0d9-515a24d69de6 2025-08-28 04:01:39,790 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/7da36e55-9df6-4d92-9053-e2bb73444592 2025-08-28 04:01:42,963 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-08-28 04:01:42,963 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-24291330 failed! 2025-08-28 04:01:42,976 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:01:47,056 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms') 2025-08-28 04:01:47,056 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-24291330 failed! 2025-08-28 04:01:47,070 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:01:51,144 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-08-28 04:01:51,144 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-24291330 failed! 2025-08-28 04:01:51,157 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:01:55,242 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1027ms') 2025-08-28 04:01:55,242 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-24291330 failed! 2025-08-28 04:01:55,256 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:01:59,345 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-08-28 04:01:59,345 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-24291330 failed! 2025-08-28 04:01:59,360 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:02:03,435 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-08-28 04:02:03,435 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-24291330 failed! 2025-08-28 04:02:03,450 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:02:07,534 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-08-28 04:02:07,534 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-24291330 failed! 2025-08-28 04:02:07,547 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:02:11,631 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-08-28 04:02:11,631 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-24291330 failed! 2025-08-28 04:02:11,644 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:02:13,705 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=6.33 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.42 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.421/3.873/6.325/2.452 ms') 2025-08-28 04:02:13,705 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-24291330 passed 2025-08-28 04:02:13,770 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:02:13,770 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-24291330, IP 192.168.1.252, Port 22 2025-08-28 04:02:13,837 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-28 04:02:13,924 - DEBUG - VM ctest-vn1_vm1-24291330 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-28 04:02:18,925 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:02:18,925 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-24291330, IP 192.168.1.252, Port 22 2025-08-28 04:02:18,992 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-28 04:02:19,078 - DEBUG - VM ctest-vn1_vm1-24291330 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-28 04:02:24,079 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:02:24,080 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-24291330, IP 192.168.1.252, Port 22 2025-08-28 04:02:24,134 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-28 04:02:24,218 - DEBUG - VM ctest-vn1_vm1-24291330 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-28 04:02:29,219 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:02:29,219 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-24291330, IP 192.168.1.252, Port 22 2025-08-28 04:02:29,375 - DEBUG - VM ctest-vn1_vm1-24291330 is ready for SSH connections 2025-08-28 04:02:29,375 - INFO - Waiting for VM ctest-vn2_vm1-38310879 to be up.. 2025-08-28 04:02:29,459 - DEBUG - VM is in ACTIVE state now 2025-08-28 04:02:29,460 - INFO - VM name : ctest-vn2_vm1-38310879 2025-08-28 04:02:29,558 - DEBUG - VM ctest-vn2_vm1-38310879 ID is 88234580-5205-418c-80bd-afb2ccb67708 2025-08-28 04:02:29,558 - DEBUG - VM ctest-vn2_vm1-38310879 launched on Node an-jenkins-deploy-platform-ansible-os-3956-2 2025-08-28 04:02:29,643 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/88234580-5205-418c-80bd-afb2ccb67708 2025-08-28 04:02:29,653 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/2078c038-2384-4d57-bfd5-c7f819cbc5df 2025-08-28 04:02:30,801 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=1.98 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.855 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.855/1.416/1.977/0.561 ms') 2025-08-28 04:02:30,802 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm1-38310879 passed 2025-08-28 04:02:30,873 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:02:30,873 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-38310879, IP 192.168.2.252, Port 22 2025-08-28 04:02:31,030 - DEBUG - VM ctest-vn2_vm1-38310879 is ready for SSH connections 2025-08-28 04:02:31,030 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:02:31,030 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.252, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.65, gateway password: c0ntrail123 2025-08-28 04:02:31,030 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.252 2025-08-28 04:02:52,114 - DEBUG - PING 192.168.2.252 (192.168.2.252) 56(84) bytes of data. 64 bytes from 192.168.2.252: icmp_req=1 ttl=63 time=8.28 ms 64 bytes from 192.168.2.252: icmp_req=2 ttl=63 time=1.27 ms 64 bytes from 192.168.2.252: icmp_req=3 ttl=63 time=2.02 ms --- 192.168.2.252 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.276/3.859/8.282/3.142 ms 2025-08-28 04:02:52,114 - INFO - Ping to IP 192.168.2.252 from VM ctest-vn1_vm1-24291330 passed 2025-08-28 04:02:55,517 - INFO - 10.0.0.65 2025-08-28 04:02:55,521 - INFO - agent:initializing 2025-08-28 04:02:55,521 - DEBUG - defaultdict(, {'10.0.0.65': {'agent': {'status': 'initializing', 'description': None}}}) 2025-08-28 04:02:55,521 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-08-28 04:03:00,610 - INFO - 10.0.0.65 2025-08-28 04:03:00,615 - INFO - agent:active 2025-08-28 04:03:00,616 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.65 2025-08-28 04:03:03,999 - INFO - 10.0.0.27 2025-08-28 04:03:04,002 - INFO - agent:initializing 2025-08-28 04:03:04,003 - DEBUG - defaultdict(, {'10.0.0.27': {'agent': {'status': 'initializing', 'description': None}}}) 2025-08-28 04:03:04,003 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-08-28 04:03:09,093 - INFO - 10.0.0.27 2025-08-28 04:03:09,098 - INFO - agent:active 2025-08-28 04:03:09,098 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.27 2025-08-28 04:03:09,800 - INFO - 10.0.0.19 2025-08-28 04:03:09,805 - INFO - control:initializing 2025-08-28 04:03:09,805 - DEBUG - defaultdict(, {'10.0.0.19': {'control': {'status': 'initializing', 'description': None}}}) 2025-08-28 04:03:09,805 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-08-28 04:03:14,910 - INFO - 10.0.0.19 2025-08-28 04:03:14,915 - INFO - control:active 2025-08-28 04:03:14,915 - INFO - Contrail services ['control'] are up on nodes 10.0.0.19 2025-08-28 04:03:15,479 - INFO - 10.0.0.38 2025-08-28 04:03:15,482 - INFO - control:initializing 2025-08-28 04:03:15,482 - DEBUG - defaultdict(, {'10.0.0.38': {'control': {'status': 'initializing', 'description': None}}}) 2025-08-28 04:03:15,482 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-08-28 04:03:20,585 - INFO - 10.0.0.38 2025-08-28 04:03:20,591 - INFO - control:active 2025-08-28 04:03:20,591 - INFO - Contrail services ['control'] are up on nodes 10.0.0.38 2025-08-28 04:03:21,083 - INFO - 10.0.0.41 2025-08-28 04:03:21,087 - INFO - control:initializing 2025-08-28 04:03:21,087 - DEBUG - defaultdict(, {'10.0.0.41': {'control': {'status': 'initializing', 'description': None}}}) 2025-08-28 04:03:21,087 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-08-28 04:03:26,185 - INFO - 10.0.0.41 2025-08-28 04:03:26,190 - INFO - control:active 2025-08-28 04:03:26,190 - INFO - Contrail services ['control'] are up on nodes 10.0.0.41 2025-08-28 04:03:26,932 - INFO - 10.0.0.19 2025-08-28 04:03:26,935 - INFO - api-server:initializing 2025-08-28 04:03:26,935 - DEBUG - defaultdict(, {'10.0.0.19': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-08-28 04:03:26,935 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-08-28 04:03:32,044 - INFO - 10.0.0.19 2025-08-28 04:03:32,047 - INFO - api-server:initializing 2025-08-28 04:03:32,048 - DEBUG - defaultdict(, {'10.0.0.19': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-08-28 04:03:32,048 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-08-28 04:03:37,211 - INFO - 10.0.0.19 2025-08-28 04:03:37,217 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-08-28 04:03:37,217 - DEBUG - defaultdict(, {'10.0.0.19': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-08-28 04:03:37,217 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-08-28 04:03:42,318 - INFO - 10.0.0.19 2025-08-28 04:03:42,322 - INFO - api-server:active 2025-08-28 04:03:42,322 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.19 2025-08-28 04:03:43,014 - INFO - 10.0.0.38 2025-08-28 04:03:43,018 - INFO - api-server:initializing 2025-08-28 04:03:43,018 - DEBUG - defaultdict(, {'10.0.0.38': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-08-28 04:03:43,018 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-08-28 04:03:48,119 - INFO - 10.0.0.38 2025-08-28 04:03:48,123 - INFO - api-server:initializing 2025-08-28 04:03:48,123 - DEBUG - defaultdict(, {'10.0.0.38': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-08-28 04:03:48,123 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-08-28 04:03:53,224 - INFO - 10.0.0.38 2025-08-28 04:03:53,230 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-08-28 04:03:53,230 - DEBUG - defaultdict(, {'10.0.0.38': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-08-28 04:03:53,230 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-08-28 04:03:58,342 - INFO - 10.0.0.38 2025-08-28 04:03:58,348 - INFO - api-server:active 2025-08-28 04:03:58,348 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.38 2025-08-28 04:03:58,996 - INFO - 10.0.0.41 2025-08-28 04:03:58,999 - INFO - api-server:initializing 2025-08-28 04:03:58,999 - DEBUG - defaultdict(, {'10.0.0.41': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-08-28 04:03:58,999 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0 2025-08-28 04:04:04,086 - INFO - 10.0.0.41 2025-08-28 04:04:04,089 - INFO - api-server:initializing 2025-08-28 04:04:04,089 - DEBUG - defaultdict(, {'10.0.0.41': {'api-server': {'status': 'initializing', 'description': None}}}) 2025-08-28 04:04:04,089 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1 2025-08-28 04:04:09,178 - INFO - 10.0.0.41 2025-08-28 04:04:09,184 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down) 2025-08-28 04:04:09,184 - DEBUG - defaultdict(, {'10.0.0.41': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}}) 2025-08-28 04:04:09,184 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2 2025-08-28 04:04:14,275 - INFO - 10.0.0.41 2025-08-28 04:04:14,280 - INFO - api-server:active 2025-08-28 04:04:14,280 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.41 2025-08-28 04:04:14,378 - INFO - 10.0.0.19 2025-08-28 04:04:14,378 - INFO - config 2025-08-28 04:04:14,383 - INFO - config-nodemgr:active 2025-08-28 04:04:14,386 - INFO - api-server:active 2025-08-28 04:04:14,390 - INFO - schema:active 2025-08-28 04:04:14,393 - INFO - svc-monitor:backup 2025-08-28 04:04:14,396 - INFO - device-manager:backup 2025-08-28 04:04:14,396 - INFO - config-database 2025-08-28 04:04:14,396 - INFO - config-cassandra:active 2025-08-28 04:04:14,396 - INFO - config-zookeeper:active 2025-08-28 04:04:14,396 - INFO - config-rabbitmq:active 2025-08-28 04:04:14,396 - INFO - control 2025-08-28 04:04:14,400 - INFO - control-nodemgr:active 2025-08-28 04:04:14,405 - INFO - control:active 2025-08-28 04:04:14,406 - INFO - named:active 2025-08-28 04:04:14,409 - INFO - dns:active 2025-08-28 04:04:14,409 - INFO - analytics 2025-08-28 04:04:14,414 - INFO - analytics-nodemgr:active 2025-08-28 04:04:14,417 - INFO - analytics-api:active 2025-08-28 04:04:14,421 - INFO - collector:active 2025-08-28 04:04:14,421 - INFO - analytics-database 2025-08-28 04:04:14,421 - INFO - analytics-cassandra:active 2025-08-28 04:04:14,426 - INFO - analyticsdb-nodemgr:active 2025-08-28 04:04:14,430 - INFO - query-engine:active 2025-08-28 04:04:14,430 - INFO - webui 2025-08-28 04:04:14,430 - INFO - webui:active 2025-08-28 04:04:14,430 - INFO - webui-middleware:active 2025-08-28 04:04:14,430 - INFO - redis:active 2025-08-28 04:04:14,430 - INFO - analytics_snmp 2025-08-28 04:04:14,434 - INFO - snmp-collector:active 2025-08-28 04:04:14,434 - INFO - snmp-topology:active 2025-08-28 04:04:14,434 - INFO - analytics_alarm 2025-08-28 04:04:14,434 - INFO - alarmgen:active 2025-08-28 04:04:14,531 - INFO - 10.0.0.38 2025-08-28 04:04:14,531 - INFO - config 2025-08-28 04:04:14,537 - INFO - config-nodemgr:active 2025-08-28 04:04:14,543 - INFO - api-server:active 2025-08-28 04:04:14,546 - INFO - schema:backup 2025-08-28 04:04:14,549 - INFO - svc-monitor:backup 2025-08-28 04:04:14,553 - INFO - device-manager:backup 2025-08-28 04:04:14,553 - INFO - config-database 2025-08-28 04:04:14,553 - INFO - config-cassandra:active 2025-08-28 04:04:14,553 - INFO - config-zookeeper:active 2025-08-28 04:04:14,553 - INFO - config-rabbitmq:active 2025-08-28 04:04:14,553 - INFO - control 2025-08-28 04:04:14,559 - INFO - control-nodemgr:active 2025-08-28 04:04:14,564 - INFO - control:active 2025-08-28 04:04:14,564 - INFO - named:active 2025-08-28 04:04:14,568 - INFO - dns:active 2025-08-28 04:04:14,568 - INFO - analytics 2025-08-28 04:04:14,573 - INFO - analytics-nodemgr:active 2025-08-28 04:04:14,577 - INFO - analytics-api:active 2025-08-28 04:04:14,582 - INFO - collector:active 2025-08-28 04:04:14,583 - INFO - analytics-database 2025-08-28 04:04:14,583 - INFO - analytics-cassandra:active 2025-08-28 04:04:14,588 - INFO - analyticsdb-nodemgr:active 2025-08-28 04:04:14,593 - INFO - query-engine:active 2025-08-28 04:04:14,593 - INFO - webui 2025-08-28 04:04:14,593 - INFO - webui:active 2025-08-28 04:04:14,593 - INFO - webui-middleware:active 2025-08-28 04:04:14,593 - INFO - redis:active 2025-08-28 04:04:14,593 - INFO - analytics_snmp 2025-08-28 04:04:14,597 - INFO - snmp-collector:active 2025-08-28 04:04:14,597 - INFO - snmp-topology:active 2025-08-28 04:04:14,598 - INFO - analytics_alarm 2025-08-28 04:04:14,598 - INFO - alarmgen:active 2025-08-28 04:04:14,686 - INFO - 10.0.0.41 2025-08-28 04:04:14,686 - INFO - config 2025-08-28 04:04:14,692 - INFO - config-nodemgr:active 2025-08-28 04:04:14,696 - INFO - api-server:active 2025-08-28 04:04:14,699 - INFO - schema:backup 2025-08-28 04:04:14,703 - INFO - svc-monitor:active 2025-08-28 04:04:14,706 - INFO - device-manager:active 2025-08-28 04:04:14,706 - INFO - config-database 2025-08-28 04:04:14,706 - INFO - config-cassandra:active 2025-08-28 04:04:14,707 - INFO - config-zookeeper:active 2025-08-28 04:04:14,707 - INFO - config-rabbitmq:active 2025-08-28 04:04:14,707 - INFO - control 2025-08-28 04:04:14,711 - INFO - control-nodemgr:active 2025-08-28 04:04:14,715 - INFO - control:active 2025-08-28 04:04:14,715 - INFO - named:active 2025-08-28 04:04:14,719 - INFO - dns:active 2025-08-28 04:04:14,719 - INFO - analytics 2025-08-28 04:04:14,723 - INFO - analytics-nodemgr:active 2025-08-28 04:04:14,727 - INFO - analytics-api:active 2025-08-28 04:04:14,732 - INFO - collector:active 2025-08-28 04:04:14,732 - INFO - analytics-database 2025-08-28 04:04:14,732 - INFO - analytics-cassandra:active 2025-08-28 04:04:14,737 - INFO - analyticsdb-nodemgr:active 2025-08-28 04:04:14,741 - INFO - query-engine:active 2025-08-28 04:04:14,741 - INFO - webui 2025-08-28 04:04:14,742 - INFO - webui:active 2025-08-28 04:04:14,742 - INFO - webui-middleware:active 2025-08-28 04:04:14,742 - INFO - redis:active 2025-08-28 04:04:14,742 - INFO - analytics_snmp 2025-08-28 04:04:14,745 - INFO - snmp-collector:active 2025-08-28 04:04:14,745 - INFO - snmp-topology:active 2025-08-28 04:04:14,745 - INFO - analytics_alarm 2025-08-28 04:04:14,745 - INFO - alarmgen:active 2025-08-28 04:04:14,834 - INFO - 10.0.0.65 2025-08-28 04:04:14,834 - INFO - vrouter 2025-08-28 04:04:14,839 - INFO - vrouter-nodemgr:active 2025-08-28 04:04:14,844 - INFO - agent:active 2025-08-28 04:04:14,933 - INFO - 10.0.0.27 2025-08-28 04:04:14,933 - INFO - vrouter 2025-08-28 04:04:14,941 - INFO - vrouter-nodemgr:active 2025-08-28 04:04:14,945 - INFO - agent:active 2025-08-28 04:04:14,945 - INFO - All the contrail services are up on all nodes 2025-08-28 04:04:14,945 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:04:15,336 - INFO - an-jenkins-deploy-platform-ansible-os-3956-1:contrail-vrouter-agent module connection to ['10.20.0.25:5269'] servers UP 2025-08-28 04:04:15,336 - INFO - an-jenkins-deploy-platform-ansible-os-3956-1 connected to xmpp 10.20.0.25 2025-08-28 04:04:15,336 - WARNING - an-jenkins-deploy-platform-ansible-os-3956-1:contrail-vrouter-agent module connection to ['10.20.0.17:5269'] servers NOT UP 2025-08-28 04:04:15,336 - INFO - an-jenkins-deploy-platform-ansible-os-3956-1:contrail-vrouter-agent module connection to ['10.20.0.254:5269'] servers UP 2025-08-28 04:04:15,336 - INFO - an-jenkins-deploy-platform-ansible-os-3956-1 connected to xmpp 10.20.0.254 2025-08-28 04:04:15,336 - INFO - an-jenkins-deploy-platform-ansible-os-3956-1:contrail-vrouter-agent module connection to ['10.20.0.25:53'] servers UP 2025-08-28 04:04:15,336 - INFO - an-jenkins-deploy-platform-ansible-os-3956-1 connected to dns 10.20.0.25 2025-08-28 04:04:15,336 - INFO - an-jenkins-deploy-platform-ansible-os-3956-1:contrail-vrouter-agent module connection to ['10.20.0.17:53'] servers UP 2025-08-28 04:04:15,337 - INFO - an-jenkins-deploy-platform-ansible-os-3956-1 connected to dns 10.20.0.17 2025-08-28 04:04:15,337 - WARNING - an-jenkins-deploy-platform-ansible-os-3956-1:contrail-vrouter-agent module connection to ['10.20.0.254:53'] servers NOT UP 2025-08-28 04:04:15,337 - WARNING - an-jenkins-deploy-platform-ansible-os-3956-1:contrail-vrouter-agent module connection to ['10.0.0.19:8086'] servers NOT UP 2025-08-28 04:04:15,337 - WARNING - an-jenkins-deploy-platform-ansible-os-3956-1:contrail-vrouter-agent module connection to ['10.0.0.38:8086'] servers NOT UP 2025-08-28 04:04:15,337 - INFO - an-jenkins-deploy-platform-ansible-os-3956-1:contrail-vrouter-agent module connection to ['10.0.0.41:8086'] servers UP 2025-08-28 04:04:15,337 - INFO - an-jenkins-deploy-platform-ansible-os-3956-1 connected to collector 10.0.0.41 2025-08-28 04:04:15,337 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:04:15,721 - INFO - cn-jenkins-deploy-platform-ansible-os-3956-1:contrail-control module connection to 10.0.0.19:9041 servers UP 2025-08-28 04:04:15,721 - INFO - cn-jenkins-deploy-platform-ansible-os-3956-1:contrail-control module connection to 10.0.0.19:5673 servers UP 2025-08-28 04:04:15,721 - INFO - cn-jenkins-deploy-platform-ansible-os-3956-1:contrail-control module connection to ['10.0.0.19:8086'] servers UP 2025-08-28 04:04:15,721 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3956-2?flat 2025-08-28 04:04:16,095 - INFO - cn-jenkins-deploy-platform-ansible-os-3956-2:contrail-control module connection to 10.0.0.19:9041 servers UP 2025-08-28 04:04:16,095 - INFO - cn-jenkins-deploy-platform-ansible-os-3956-2:contrail-control module connection to 10.0.0.19:5673 servers UP 2025-08-28 04:04:16,096 - INFO - cn-jenkins-deploy-platform-ansible-os-3956-2:contrail-control module connection to ['10.0.0.19:8086'] servers UP 2025-08-28 04:04:16,096 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3956-3?flat 2025-08-28 04:04:16,114 - INFO - cn-jenkins-deploy-platform-ansible-os-3956-3:contrail-control module connection to 10.0.0.19:9041 servers UP 2025-08-28 04:04:16,114 - INFO - cn-jenkins-deploy-platform-ansible-os-3956-3:contrail-control module connection to 10.0.0.19:5673 servers UP 2025-08-28 04:04:16,115 - INFO - cn-jenkins-deploy-platform-ansible-os-3956-3:contrail-control module connection to ['10.0.0.19:8086'] servers UP 2025-08-28 04:04:16,115 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3956-1.?flat 2025-08-28 04:04:16,137 - DEBUG - contrail-api is running 2025-08-28 04:04:16,137 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3956-2.?flat 2025-08-28 04:04:16,158 - DEBUG - contrail-api is running 2025-08-28 04:04:16,158 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3956-3.?flat 2025-08-28 04:04:16,182 - DEBUG - contrail-api is running 2025-08-28 04:04:16,473 - INFO - Created VN ctest-vn3-22907884, UUID :61d8f4a0-d75c-4f15-92e1-d9f2720ce4ef 2025-08-28 04:04:16,908 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 04:04:16,943 - DEBUG - Requesting: http://10.0.0.19:8082/domain/692c5a68-a199-4232-a07c-1cd4918de9d4 2025-08-28 04:04:16,956 - DEBUG - Requesting: http://10.0.0.19:8082/project/5631e639-3d56-4dbe-9639-2fd2c3fa5ac2 2025-08-28 04:04:17,046 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/b897aa08-efeb-466e-89f1-72b7f9200eae 2025-08-28 04:04:17,075 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/b897aa08-efeb-466e-89f1-72b7f9200eae 2025-08-28 04:04:17,092 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/36da7fbc-411a-4bba-ada9-874f8db9143c 2025-08-28 04:04:17,108 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/36da7fbc-411a-4bba-ada9-874f8db9143c 2025-08-28 04:04:17,119 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/80ee3fea-82cb-4a45-8e77-aa1b8e938f4b 2025-08-28 04:04:17,132 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 04:04:17,132 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/b897aa08-efeb-466e-89f1-72b7f9200eae 2025-08-28 04:04:17,143 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/36da7fbc-411a-4bba-ada9-874f8db9143c 2025-08-28 04:04:17,160 - INFO - Verified VN network id 12 for VN b897aa08-efeb-466e-89f1-72b7f9200eae 2025-08-28 04:04:17,160 - INFO - Verifications in API Server for VN ctest-vn1-50579577 passed 2025-08-28 04:04:17,160 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/b897aa08-efeb-466e-89f1-72b7f9200eae 2025-08-28 04:04:17,171 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/36da7fbc-411a-4bba-ada9-874f8db9143c 2025-08-28 04:04:17,179 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/36da7fbc-411a-4bba-ada9-874f8db9143c 2025-08-28 04:04:17,188 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/80ee3fea-82cb-4a45-8e77-aa1b8e938f4b 2025-08-28 04:04:17,202 - DEBUG - Control-node 10.0.0.19 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-72692947', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13301286979823683182', 'uuid-lslong': '9939851987067604654'}, 'enable': 'true', 'created': '2025-08-28T04:01:29', 'last-modified': '2025-08-28T04:01:30', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:06.908580'} 2025-08-28 04:04:17,209 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 04:04:17,215 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-72692947', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13301286979823683182', 'uuid-lslong': '9939851987067604654'}, 'enable': 'true', 'created': '2025-08-28T04:01:29', 'last-modified': '2025-08-28T04:01:30', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:01.312586'} 2025-08-28 04:04:17,221 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 04:04:17,228 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-72692947', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13301286979823683182', 'uuid-lslong': '9939851987067604654'}, 'enable': 'true', 'created': '2025-08-28T04:01:29', 'last-modified': '2025-08-28T04:01:30', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:55.729532'} 2025-08-28 04:04:17,234 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-28 04:04:17,235 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-50579577 passed 2025-08-28 04:04:17,235 - DEBUG - ====Verifying policy data for ctest-vn1-50579577 in API_Server ====== 2025-08-28 04:04:17,235 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 04:04:17,241 - DEBUG - Requesting: http://10.0.0.19:8082/domain/692c5a68-a199-4232-a07c-1cd4918de9d4 2025-08-28 04:04:17,253 - DEBUG - Requesting: http://10.0.0.19:8082/project/5631e639-3d56-4dbe-9639-2fd2c3fa5ac2 2025-08-28 04:04:17,352 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/b897aa08-efeb-466e-89f1-72b7f9200eae 2025-08-28 04:04:17,363 - DEBUG - ==>Verifying data for policy with id: 9a37916c-d056-4a03-9356-8bb3fcc7c669, fqn: ['default-domain', 'ctest-TestBasicVMVN0-72692947', 'policy1'] 2025-08-28 04:04:17,364 - INFO - VN ctest-vn1-50579577 Policy verification: verify_vn_policy_in_api_server, status: True 2025-08-28 04:04:17,364 - DEBUG - Verifying the vn in opserver 2025-08-28 04:04:17,364 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 virtual network link through opserver 10.0.0.19 2025-08-28 04:04:17,364 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2025-08-28 04:04:17,376 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577?flat'} 2025-08-28 04:04:17,376 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 virtual network link through opserver 10.0.0.38 2025-08-28 04:04:17,376 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-28 04:04:17,434 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577?flat'} 2025-08-28 04:04:17,434 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 virtual network link through opserver 10.0.0.41 2025-08-28 04:04:17,434 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-08-28 04:04:17,489 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577?flat'} 2025-08-28 04:04:17,489 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is found in opserver 2025-08-28 04:04:17,508 - DEBUG - VRF ids for VN ctest-vn1-50579577: {'10.0.0.65': '2'} 2025-08-28 04:04:18,827 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3956-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3956-1) 2025-08-28 04:04:18,875 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-08-28 04:04:23,957 - DEBUG - VM is in ACTIVE state now 2025-08-28 04:04:23,957 - INFO - VM name : ctest-vn1_vm2-56544639 2025-08-28 04:04:24,032 - DEBUG - VM ctest-vn1_vm2-56544639 ID is de9f8549-f809-4a1e-ad57-10a1afb79178 2025-08-28 04:04:24,120 - DEBUG - VM is in ACTIVE state now 2025-08-28 04:04:24,120 - DEBUG - VM ctest-vn1_vm2-56544639 launched on Node an-jenkins-deploy-platform-ansible-os-3956-1 2025-08-28 04:04:24,201 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178 2025-08-28 04:04:24,232 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178 2025-08-28 04:04:24,239 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/71fdb443-ebbc-461f-9e84-5f1d4ec2ada4 2025-08-28 04:04:24,247 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178 2025-08-28 04:04:24,254 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/71fdb443-ebbc-461f-9e84-5f1d4ec2ada4 2025-08-28 04:04:24,263 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/96daec55-6abc-46d1-90b2-59fcf85a781d 2025-08-28 04:04:24,271 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 04:04:24,271 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178 2025-08-28 04:04:24,277 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/71fdb443-ebbc-461f-9e84-5f1d4ec2ada4 2025-08-28 04:04:24,287 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/96daec55-6abc-46d1-90b2-59fcf85a781d 2025-08-28 04:04:24,294 - INFO - VM ctest-vn1_vm2-56544639 verfication in all API Servers passed 2025-08-28 04:04:24,310 - DEBUG - VM ctest-vn1_vm2-56544639 Tap interface: {'index': '4', 'name': 'tap71fdb443-eb', 'uuid': '71fdb443-ebbc-461f-9e84-5f1d4ec2ada4', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577:ctest-vn1-50579577', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'vm_uuid': 'de9f8549-f809-4a1e-ad57-10a1afb79178', 'vm_name': 'ctest-vn1_vm2-56544639', 'ip_addr': '192.168.1.251', 'mac_addr': '02:71:fd:b4:43:eb', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '164', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-72692947:71fdb443-ebbc-461f-9e84-5f1d4ec2ada4', 'sg_uuid_list': ['9e2e8ff8-0b3f-400e-973b-a630f0dafd7f'], 'static_route_list': None, 'vm_project_uuid': '5631e639-3d56-4dbe-9639-2fd2c3fa5ac2', 'admin_state': 'Enabled', 'flow_key_idx': '31', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-28 04:04:24,310 - DEBUG - Agent 10.0.0.65 vrf name: default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577:ctest-vn1-50579577 2025-08-28 04:04:24,316 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577:ctest-vn1-50579577', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'table_label': '-1', 'vxlan_id': '12', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.193:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-08-28 04:04:24,396 - DEBUG - VM ctest-vn1_vm2-56544639 : Tap interface tap71fdb443-eb is set to Active 2025-08-28 04:04:24,397 - DEBUG - Tap interface tap71fdb443-eb detail : {'index': '4', 'name': 'tap71fdb443-eb', 'uuid': '71fdb443-ebbc-461f-9e84-5f1d4ec2ada4', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577:ctest-vn1-50579577', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'vm_uuid': 'de9f8549-f809-4a1e-ad57-10a1afb79178', 'vm_name': 'ctest-vn1_vm2-56544639', 'ip_addr': '192.168.1.251', 'mac_addr': '02:71:fd:b4:43:eb', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '164', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-72692947:71fdb443-ebbc-461f-9e84-5f1d4ec2ada4', 'sg_uuid_list': ['9e2e8ff8-0b3f-400e-973b-a630f0dafd7f'], 'static_route_list': None, 'vm_project_uuid': '5631e639-3d56-4dbe-9639-2fd2c3fa5ac2', 'admin_state': 'Enabled', 'flow_key_idx': '31', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-28 04:04:24,505 - DEBUG - Starting Layer 2 verification in Agent 2025-08-28 04:04:24,518 - DEBUG - Layer 2 path is seen for VM MAC 02:71:fd:b4:43:eb in agent 10.0.0.65 2025-08-28 04:04:24,518 - DEBUG - Active layer 2 route in agent is present for VMI tap71fdb443-eb 2025-08-28 04:04:24,518 - DEBUG - L2 label(34) matches bw route table and interface table 2025-08-28 04:04:24,529 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is found in Agent of node 10.0.0.65 2025-08-28 04:04:24,654 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is consistent in agent 10.0.0.65 2025-08-28 04:04:24,655 - DEBUG - Route for VM IP ['192.168.1.251'] is consistent in agent 10.0.0.65 2025-08-28 04:04:24,655 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 verification for VM ctest-vn1_vm2-56544639 in Agent 10.0.0.65 passed 2025-08-28 04:04:24,761 - DEBUG - Starting all layer 2 verification in agent 10.0.0.65 2025-08-28 04:04:24,774 - DEBUG - Route for VM MAC 02:71:fd:b4:43:eb is consistent in agent 10.0.0.65 2025-08-28 04:04:27,981 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1031ms') 2025-08-28 04:04:27,981 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-56544639 failed! 2025-08-28 04:04:27,998 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:04:32,076 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-08-28 04:04:32,076 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-56544639 failed! 2025-08-28 04:04:32,091 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:04:36,173 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-08-28 04:04:36,173 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-56544639 failed! 2025-08-28 04:04:36,196 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:04:40,270 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms') 2025-08-28 04:04:40,270 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-56544639 failed! 2025-08-28 04:04:40,285 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:04:44,373 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-08-28 04:04:44,373 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-56544639 failed! 2025-08-28 04:04:44,389 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:04:48,465 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms') 2025-08-28 04:04:48,465 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-56544639 failed! 2025-08-28 04:04:48,478 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:04:52,559 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-08-28 04:04:52,559 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-56544639 failed! 2025-08-28 04:04:52,573 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 is 192.168.1.254 and allocation pool is NOT set 2025-08-28 04:04:56,649 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=7.73 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1019ms\r\nrtt min/avg/max/mdev = 7.729/7.729/7.729/0.000 ms') 2025-08-28 04:04:56,649 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-56544639 passed 2025-08-28 04:04:56,649 - INFO - VM ctest-vn1_vm2-56544639 verifications in Compute nodes passed 2025-08-28 04:04:56,844 - DEBUG - Validated VM route 192.168.1.251 in vrouter of 10.0.0.65 2025-08-28 04:04:56,849 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-08-28 04:04:56,849 - INFO - Validated routes of VM ctest-vn1_vm2-56544639 in all vrouters 2025-08-28 04:04:57,287 - DEBUG - Starting all layer2 verification in 10.0.0.41 Control Node 2025-08-28 04:04:57,410 - DEBUG - Layer2 route found for VM MAC 02:71:fd:b4:43:eb in Control-node 10.0.0.41 2025-08-28 04:04:57,410 - DEBUG - L2 Label for VM ctest-vn1_vm2-56544639 same between Control-node 10.0.0.41 and Agent, Expected: 34, Seen: 34 2025-08-28 04:04:57,410 - DEBUG - Starting all layer2 verification in 10.0.0.19 Control Node 2025-08-28 04:04:57,534 - DEBUG - Layer2 route found for VM MAC 02:71:fd:b4:43:eb in Control-node 10.0.0.19 2025-08-28 04:04:57,534 - DEBUG - L2 Label for VM ctest-vn1_vm2-56544639 same between Control-node 10.0.0.19 and Agent, Expected: 34, Seen: 34 2025-08-28 04:04:57,534 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-56544639 passed 2025-08-28 04:04:57,534 - DEBUG - Verifying the vm in opserver 2025-08-28 04:04:57,534 - DEBUG - Verifying in collector 10.0.0.41 ... 2025-08-28 04:04:57,534 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178?flat 2025-08-28 04:04:57,594 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-72692947:71fdb443-ebbc-461f-9e84-5f1d4ec2ada4?flat 2025-08-28 04:04:57,611 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-72692947:71fdb443-ebbc-461f-9e84-5f1d4ec2ada4?flat 2025-08-28 04:04:57,628 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178?flat 2025-08-28 04:04:57,641 - DEBUG - Verifying vm in vn uve 2025-08-28 04:04:57,641 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-72692947:71fdb443-ebbc-461f-9e84-5f1d4ec2ada4?flat 2025-08-28 04:04:57,658 - DEBUG - VM uve shows interface as {'ip6_address': '::', 'admin_state': True, '__T': 1756353866081612, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577', 'tx_vlan': 65535, 'label': 30, 'vm_uuid': 'de9f8549-f809-4a1e-ad57-10a1afb79178', 'active': True, 'gateway': '192.168.1.254', 'rx_vlan': 65535, 'uuid': '71fdb443-ebbc-461f-9e84-5f1d4ec2ada4', 'is_health_check_active': True, 'port_mirror_enabled': False, 'fixed_ip4_list': ['192.168.1.251'], 'vhostuser_mode': 0, 'ip_address': '192.168.1.251', 'ip6_active': False, 'vn_uuid': 'b897aa08-efeb-466e-89f1-72b7f9200eae', 'l2_active': True, 'ip4_active': True, 'vm_name': 'ctest-vn1_vm2-56544639', 'mac_address': '02:71:fd:b4:43:eb'} 2025-08-28 04:04:57,658 - DEBUG - VM uve shows ip address as ['192.168.1.251', '::'] 2025-08-28 04:04:57,658 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 2025-08-28 04:04:57,658 - INFO - Verifying through opserver in 10.0.0.19 2025-08-28 04:04:57,658 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577?flat 2025-08-28 04:04:57,679 - DEBUG - expected vm list de9f8549-f809-4a1e-ad57-10a1afb79178 2025-08-28 04:04:57,679 - DEBUG - Extracted vm list ['de9f8549-f809-4a1e-ad57-10a1afb79178'] 2025-08-28 04:04:57,679 - INFO - VM de9f8549-f809-4a1e-ad57-10a1afb79178 is present in default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 2025-08-28 04:04:57,679 - DEBUG - Verifying vm in vrouter uve 2025-08-28 04:04:57,679 - DEBUG - Getting info from collector 10.0.0.41.. 2025-08-28 04:04:57,679 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178?flat 2025-08-28 04:04:57,737 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-3956-1'] 2025-08-28 04:04:57,737 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:04:57,793 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:04:57,821 - DEBUG - VM de9f8549-f809-4a1e-ad57-10a1afb79178 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:04:57,821 - DEBUG - Validated that VM ctest-vn1_vm2-56544639 is in Vrouter an-jenkins-deploy-platform-ansible-os-3956-1 UVE 2025-08-28 04:04:57,821 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-08-28 04:04:57,829 - DEBUG - Expected tap interface of VM uuid de9f8549-f809-4a1e-ad57-10a1afb79178 is default-domain:ctest-TestBasicVMVN0-72692947:71fdb443-ebbc-461f-9e84-5f1d4ec2ada4 2025-08-28 04:04:57,830 - DEBUG - Expected VN of VM uuid de9f8549-f809-4a1e-ad57-10a1afb79178 is default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 2025-08-28 04:04:57,830 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:04:57,883 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:04:57,911 - DEBUG - VM de9f8549-f809-4a1e-ad57-10a1afb79178 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:04:57,911 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-72692947:71fdb443-ebbc-461f-9e84-5f1d4ec2ada4 of vm de9f8549-f809-4a1e-ad57-10a1afb79178 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:04:57,911 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 of vm de9f8549-f809-4a1e-ad57-10a1afb79178 is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:04:57,911 - DEBUG - Verifying the de9f8549-f809-4a1e-ad57-10a1afb79178 virtual network link through opserver 10.0.0.19 2025-08-28 04:04:57,911 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-08-28 04:04:57,923 - DEBUG - VM link and name as {'name': 'de9f8549-f809-4a1e-ad57-10a1afb79178', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178?flat'} 2025-08-28 04:04:57,923 - DEBUG - Verifying the de9f8549-f809-4a1e-ad57-10a1afb79178 virtual network link through opserver 10.0.0.38 2025-08-28 04:04:57,923 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-28 04:04:57,982 - DEBUG - VM link and name as {'name': 'de9f8549-f809-4a1e-ad57-10a1afb79178', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178?flat'} 2025-08-28 04:04:57,982 - DEBUG - Verifying the de9f8549-f809-4a1e-ad57-10a1afb79178 virtual network link through opserver 10.0.0.41 2025-08-28 04:04:57,982 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines 2025-08-28 04:04:57,995 - DEBUG - VM link and name as {'name': 'de9f8549-f809-4a1e-ad57-10a1afb79178', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178?flat'} 2025-08-28 04:04:57,995 - INFO - VM ctest-vn1_vm2-56544639 validations in Opserver passed 2025-08-28 04:04:59,207 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3956-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3956-2) 2025-08-28 04:04:59,255 - DEBUG - VM is still in BUILD state, Expected: ACTIVE 2025-08-28 04:05:04,335 - DEBUG - VM is in ACTIVE state now 2025-08-28 04:05:04,335 - INFO - VM name : ctest-vn2_vm2-33498639 2025-08-28 04:05:04,421 - DEBUG - VM ctest-vn2_vm2-33498639 ID is 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec 2025-08-28 04:05:04,493 - DEBUG - VM is in ACTIVE state now 2025-08-28 04:05:04,493 - DEBUG - VM ctest-vn2_vm2-33498639 launched on Node an-jenkins-deploy-platform-ansible-os-3956-2 2025-08-28 04:05:04,598 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec 2025-08-28 04:05:04,608 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec 2025-08-28 04:05:04,619 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/dd5b6ae0-4bc5-4ed4-8a7a-4e2a4b55eb26 2025-08-28 04:05:04,631 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec 2025-08-28 04:05:04,639 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/dd5b6ae0-4bc5-4ed4-8a7a-4e2a4b55eb26 2025-08-28 04:05:04,650 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/0f121603-e0b2-45f6-8c4b-c1ae1437d527 2025-08-28 04:05:04,660 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 04:05:04,660 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec 2025-08-28 04:05:04,667 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/dd5b6ae0-4bc5-4ed4-8a7a-4e2a4b55eb26 2025-08-28 04:05:04,677 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/0f121603-e0b2-45f6-8c4b-c1ae1437d527 2025-08-28 04:05:04,683 - INFO - VM ctest-vn2_vm2-33498639 verfication in all API Servers passed 2025-08-28 04:05:04,700 - DEBUG - VM ctest-vn2_vm2-33498639 Tap interface: {'index': '4', 'name': 'tapdd5b6ae0-4b', 'uuid': 'dd5b6ae0-4bc5-4ed4-8a7a-4e2a4b55eb26', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093:ctest-vn2-66914093', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093', 'vm_uuid': '59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec', 'vm_name': 'ctest-vn2_vm2-33498639', 'ip_addr': '192.168.2.251', 'mac_addr': '02:dd:5b:6a:e0:4b', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '131', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-72692947:dd5b6ae0-4bc5-4ed4-8a7a-4e2a4b55eb26', 'sg_uuid_list': ['9e2e8ff8-0b3f-400e-973b-a630f0dafd7f'], 'static_route_list': None, 'vm_project_uuid': '5631e639-3d56-4dbe-9639-2fd2c3fa5ac2', 'admin_state': 'Enabled', 'flow_key_idx': '33', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-28 04:05:04,700 - DEBUG - Agent 10.0.0.27 vrf name: default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093:ctest-vn2-66914093 2025-08-28 04:05:04,705 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093:ctest-vn2-66914093', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.192:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]} 2025-08-28 04:05:04,780 - DEBUG - VM ctest-vn2_vm2-33498639 : Tap interface tapdd5b6ae0-4b is set to Active 2025-08-28 04:05:04,781 - DEBUG - Tap interface tapdd5b6ae0-4b detail : {'index': '4', 'name': 'tapdd5b6ae0-4b', 'uuid': 'dd5b6ae0-4bc5-4ed4-8a7a-4e2a4b55eb26', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093:ctest-vn2-66914093', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093', 'vm_uuid': '59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec', 'vm_name': 'ctest-vn2_vm2-33498639', 'ip_addr': '192.168.2.251', 'mac_addr': '02:dd:5b:6a:e0:4b', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '131', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-72692947:dd5b6ae0-4bc5-4ed4-8a7a-4e2a4b55eb26', 'sg_uuid_list': ['9e2e8ff8-0b3f-400e-973b-a630f0dafd7f'], 'static_route_list': None, 'vm_project_uuid': '5631e639-3d56-4dbe-9639-2fd2c3fa5ac2', 'admin_state': 'Enabled', 'flow_key_idx': '33', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []} 2025-08-28 04:05:04,889 - DEBUG - Starting Layer 2 verification in Agent 2025-08-28 04:05:04,901 - DEBUG - Layer 2 path is seen for VM MAC 02:dd:5b:6a:e0:4b in agent 10.0.0.27 2025-08-28 04:05:04,901 - DEBUG - Active layer 2 route in agent is present for VMI tapdd5b6ae0-4b 2025-08-28 04:05:04,901 - DEBUG - L2 label(34) matches bw route table and interface table 2025-08-28 04:05:04,916 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 is found in Agent of node 10.0.0.27 2025-08-28 04:05:05,045 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 is consistent in agent 10.0.0.27 2025-08-28 04:05:05,045 - DEBUG - Route for VM IP ['192.168.2.251'] is consistent in agent 10.0.0.27 2025-08-28 04:05:05,045 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 verification for VM ctest-vn2_vm2-33498639 in Agent 10.0.0.27 passed 2025-08-28 04:05:05,153 - DEBUG - Starting all layer 2 verification in agent 10.0.0.27 2025-08-28 04:05:05,167 - DEBUG - Route for VM MAC 02:dd:5b:6a:e0:4b is consistent in agent 10.0.0.27 2025-08-28 04:05:08,351 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms') 2025-08-28 04:05:08,351 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33498639 failed! 2025-08-28 04:05:08,365 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 is 192.168.2.254 and allocation pool is NOT set 2025-08-28 04:05:12,454 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1027ms') 2025-08-28 04:05:12,454 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33498639 failed! 2025-08-28 04:05:12,469 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 is 192.168.2.254 and allocation pool is NOT set 2025-08-28 04:05:16,543 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms') 2025-08-28 04:05:16,543 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33498639 failed! 2025-08-28 04:05:16,557 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 is 192.168.2.254 and allocation pool is NOT set 2025-08-28 04:05:20,643 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1027ms') 2025-08-28 04:05:20,643 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33498639 failed! 2025-08-28 04:05:20,657 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 is 192.168.2.254 and allocation pool is NOT set 2025-08-28 04:05:24,744 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms') 2025-08-28 04:05:24,744 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33498639 failed! 2025-08-28 04:05:24,758 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 is 192.168.2.254 and allocation pool is NOT set 2025-08-28 04:05:28,833 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms') 2025-08-28 04:05:28,833 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33498639 failed! 2025-08-28 04:05:28,846 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 is 192.168.2.254 and allocation pool is NOT set 2025-08-28 04:05:32,935 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms') 2025-08-28 04:05:32,935 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33498639 failed! 2025-08-28 04:05:32,950 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 is 192.168.2.254 and allocation pool is NOT set 2025-08-28 04:05:35,015 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=7.15 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=2.76 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 2.756/4.953/7.151/2.197 ms') 2025-08-28 04:05:35,015 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33498639 passed 2025-08-28 04:05:35,016 - INFO - VM ctest-vn2_vm2-33498639 verifications in Compute nodes passed 2025-08-28 04:05:35,215 - DEBUG - Validated VM route 192.168.2.251 in vrouter of 10.0.0.27 2025-08-28 04:05:35,220 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0 2025-08-28 04:05:35,220 - INFO - Validated routes of VM ctest-vn2_vm2-33498639 in all vrouters 2025-08-28 04:05:35,636 - DEBUG - Starting all layer2 verification in 10.0.0.41 Control Node 2025-08-28 04:05:35,769 - DEBUG - Layer2 route found for VM MAC 02:dd:5b:6a:e0:4b in Control-node 10.0.0.41 2025-08-28 04:05:35,769 - DEBUG - L2 Label for VM ctest-vn2_vm2-33498639 same between Control-node 10.0.0.41 and Agent, Expected: 34, Seen: 34 2025-08-28 04:05:35,769 - DEBUG - Starting all layer2 verification in 10.0.0.19 Control Node 2025-08-28 04:05:35,895 - DEBUG - Layer2 route found for VM MAC 02:dd:5b:6a:e0:4b in Control-node 10.0.0.19 2025-08-28 04:05:35,895 - DEBUG - L2 Label for VM ctest-vn2_vm2-33498639 same between Control-node 10.0.0.19 and Agent, Expected: 34, Seen: 34 2025-08-28 04:05:35,895 - INFO - Verification in Control-nodes for VM ctest-vn2_vm2-33498639 passed 2025-08-28 04:05:35,895 - DEBUG - Verifying the vm in opserver 2025-08-28 04:05:35,895 - DEBUG - Verifying in collector 10.0.0.41 ... 2025-08-28 04:05:35,895 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec?flat 2025-08-28 04:05:35,917 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-72692947:dd5b6ae0-4bc5-4ed4-8a7a-4e2a4b55eb26?flat 2025-08-28 04:05:35,940 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-72692947:dd5b6ae0-4bc5-4ed4-8a7a-4e2a4b55eb26?flat 2025-08-28 04:05:35,958 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec?flat 2025-08-28 04:05:35,996 - DEBUG - Verifying vm in vn uve 2025-08-28 04:05:35,996 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-72692947:dd5b6ae0-4bc5-4ed4-8a7a-4e2a4b55eb26?flat 2025-08-28 04:05:36,019 - DEBUG - VM uve shows interface as {'vm_name': 'ctest-vn2_vm2-33498639', 'ip4_active': True, 'active': True, '__T': 1756353904535666, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093', 'mac_address': '02:dd:5b:6a:e0:4b', 'label': 30, 'ip6_active': False, 'ip6_address': '::', 'gateway': '192.168.2.254', 'fixed_ip4_list': ['192.168.2.251'], 'is_health_check_active': True, 'tx_vlan': 65535, 'l2_active': True, 'port_mirror_enabled': False, 'vm_uuid': '59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec', 'vn_uuid': 'c7e4e6f2-739e-41ec-9ed8-5516e9bfc536', 'ip_address': '192.168.2.251', 'admin_state': True, 'rx_vlan': 65535, 'vhostuser_mode': 0, 'uuid': 'dd5b6ae0-4bc5-4ed4-8a7a-4e2a4b55eb26'} 2025-08-28 04:05:36,019 - DEBUG - VM uve shows ip address as ['192.168.2.251', '::'] 2025-08-28 04:05:36,019 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 2025-08-28 04:05:36,019 - INFO - Verifying through opserver in 10.0.0.19 2025-08-28 04:05:36,019 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093?flat 2025-08-28 04:05:36,043 - DEBUG - expected vm list 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec 2025-08-28 04:05:36,043 - DEBUG - Extracted vm list ['59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec'] 2025-08-28 04:05:36,043 - INFO - VM 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec is present in default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 2025-08-28 04:05:36,043 - DEBUG - Verifying vm in vrouter uve 2025-08-28 04:05:36,043 - DEBUG - Getting info from collector 10.0.0.41.. 2025-08-28 04:05:36,043 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec?flat 2025-08-28 04:05:36,057 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-3956-2'] 2025-08-28 04:05:36,057 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-2:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:05:36,118 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-2?flat 2025-08-28 04:05:36,156 - DEBUG - VM 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-2 uve 2025-08-28 04:05:36,156 - DEBUG - Validated that VM ctest-vn2_vm2-33498639 is in Vrouter an-jenkins-deploy-platform-ansible-os-3956-2 UVE 2025-08-28 04:05:36,156 - DEBUG - Verifying vm tap interface/vn in vrouter uve 2025-08-28 04:05:36,166 - DEBUG - Expected tap interface of VM uuid 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec is default-domain:ctest-TestBasicVMVN0-72692947:dd5b6ae0-4bc5-4ed4-8a7a-4e2a4b55eb26 2025-08-28 04:05:36,166 - DEBUG - Expected VN of VM uuid 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec is default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 2025-08-28 04:05:36,166 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-2:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:05:36,225 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-2?flat 2025-08-28 04:05:36,257 - DEBUG - VM 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-2 uve 2025-08-28 04:05:36,257 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-72692947:dd5b6ae0-4bc5-4ed4-8a7a-4e2a4b55eb26 of vm 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-2 uve 2025-08-28 04:05:36,257 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 of vm 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec is present in vrouter an-jenkins-deploy-platform-ansible-os-3956-2 uve 2025-08-28 04:05:36,257 - DEBUG - Verifying the 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec virtual network link through opserver 10.0.0.19 2025-08-28 04:05:36,257 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-08-28 04:05:36,268 - DEBUG - VM link and name as {'name': '59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec?flat'} 2025-08-28 04:05:36,268 - DEBUG - Verifying the 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec virtual network link through opserver 10.0.0.38 2025-08-28 04:05:36,268 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-28 04:05:36,280 - DEBUG - VM link and name as {'name': '59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec?flat'} 2025-08-28 04:05:36,280 - DEBUG - Verifying the 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec virtual network link through opserver 10.0.0.41 2025-08-28 04:05:36,280 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines 2025-08-28 04:05:36,291 - DEBUG - VM link and name as {'name': '59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec?flat'} 2025-08-28 04:05:36,291 - INFO - VM ctest-vn2_vm2-33498639 validations in Opserver passed 2025-08-28 04:05:36,291 - INFO - Waiting for VM ctest-vn1_vm2-56544639 to be up.. 2025-08-28 04:05:36,291 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:05:36,291 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178 2025-08-28 04:05:36,299 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/71fdb443-ebbc-461f-9e84-5f1d4ec2ada4 2025-08-28 04:05:37,454 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=2.83 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.441 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.441/1.635/2.830/1.194 ms') 2025-08-28 04:05:37,454 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-56544639 passed 2025-08-28 04:05:37,524 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:05:37,524 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-56544639, IP 192.168.1.251, Port 22 2025-08-28 04:05:37,689 - DEBUG - VM ctest-vn1_vm2-56544639 is ready for SSH connections 2025-08-28 04:05:37,689 - INFO - Waiting for VM ctest-vn2_vm2-33498639 to be up.. 2025-08-28 04:05:37,689 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:05:37,689 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec 2025-08-28 04:05:37,697 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/dd5b6ae0-4bc5-4ed4-8a7a-4e2a4b55eb26 2025-08-28 04:05:38,863 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=2.16 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.834 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.834/1.497/2.161/0.663 ms') 2025-08-28 04:05:38,863 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-33498639 passed 2025-08-28 04:05:38,929 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:05:38,929 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-33498639, IP 192.168.2.251, Port 22 2025-08-28 04:05:38,985 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-28 04:05:39,071 - DEBUG - VM ctest-vn2_vm2-33498639 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-28 04:05:44,072 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:05:44,073 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-33498639, IP 192.168.2.251, Port 22 2025-08-28 04:05:44,130 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-28 04:05:44,207 - DEBUG - VM ctest-vn2_vm2-33498639 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-28 04:05:49,208 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:05:49,209 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-33498639, IP 192.168.2.251, Port 22 2025-08-28 04:05:49,276 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-28 04:05:49,389 - DEBUG - VM ctest-vn2_vm2-33498639 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-28 04:05:54,389 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:05:54,390 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-33498639, IP 192.168.2.251, Port 22 2025-08-28 04:05:54,545 - DEBUG - VM ctest-vn2_vm2-33498639 is ready for SSH connections 2025-08-28 04:05:54,546 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-28 04:05:54,546 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.251, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.65, gateway password: c0ntrail123 2025-08-28 04:05:54,546 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.251 2025-08-28 04:06:32,054 - DEBUG - PING 192.168.2.251 (192.168.2.251) 56(84) bytes of data. 64 bytes from 192.168.2.251: icmp_req=1 ttl=63 time=7.62 ms 64 bytes from 192.168.2.251: icmp_req=2 ttl=63 time=1.12 ms 64 bytes from 192.168.2.251: icmp_req=3 ttl=63 time=1.60 ms --- 192.168.2.251 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.124/3.451/7.626/2.959 ms 2025-08-28 04:06:32,054 - INFO - Ping to IP 192.168.2.251 from VM ctest-vn1_vm2-56544639 passed 2025-08-28 04:06:32,054 - INFO - Deleting VM ctest-vn2_vm2-33498639 2025-08-28 04:06:32,122 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 04:06:32,122 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec 2025-08-28 04:06:32,130 - DEBUG - VM ID 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec of VM ctest-vn2_vm2-33498639 is still found in API Server 2025-08-28 04:06:34,131 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 04:06:34,131 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec 2025-08-28 04:06:34,138 - DEBUG - Response Code: 404 2025-08-28 04:06:34,138 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec 2025-08-28 04:06:34,143 - DEBUG - Response Code: 404 2025-08-28 04:06:34,143 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec 2025-08-28 04:06:34,149 - DEBUG - Response Code: 404 2025-08-28 04:06:34,149 - INFO - VM ctest-vn2_vm2-33498639 is fully removed in API-Server 2025-08-28 04:06:34,304 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-08-28 04:06:34,334 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 2025-08-28 04:06:34,453 - DEBUG - VM route 192.168.2.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 of compute 10.0.0.27 2025-08-28 04:06:34,459 - DEBUG - Validated that vrouter 10.0.0.27 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 2025-08-28 04:06:34,459 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 2025-08-28 04:06:34,459 - INFO - VM ctest-vn2_vm2-33498639 is removed in Compute, and routes are removed in all compute nodes 2025-08-28 04:06:34,666 - INFO - Routes for VM ctest-vn2_vm2-33498639 is removed in all control-nodes 2025-08-28 04:06:34,666 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-2:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:06:34,722 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-2?flat 2025-08-28 04:06:34,757 - DEBUG - VM 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec is not present in vrouter an-jenkins-deploy-platform-ansible-os-3956-2 uve 2025-08-28 04:06:34,758 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:06:34,758 - DEBUG - interface for vm 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec deleted from vrouter uve ... 2025-08-28 04:06:34,758 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:06:34,758 - DEBUG - Validated that VM 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec is removed in Vrouter UVE 2025-08-28 04:06:34,758 - DEBUG - Verifying through opserver in 10.0.0.19 2025-08-28 04:06:34,758 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093?flat 2025-08-28 04:06:34,783 - DEBUG - 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec BM not in default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn2-66914093 uve 2025-08-28 04:06:34,783 - DEBUG - Validated that VM 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec is not present in the VN UVE 2025-08-28 04:06:34,783 - DEBUG - Verifying the 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec virtual network link through opserver 10.0.0.19 2025-08-28 04:06:34,783 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-08-28 04:06:34,793 - INFO - 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec vm uve deleted from opserver 2025-08-28 04:06:34,794 - DEBUG - Verifying the 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec virtual network link through opserver 10.0.0.38 2025-08-28 04:06:34,794 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-28 04:06:34,851 - INFO - 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec vm uve deleted from opserver 2025-08-28 04:06:34,851 - DEBUG - Verifying the 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec virtual network link through opserver 10.0.0.41 2025-08-28 04:06:34,851 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines 2025-08-28 04:06:34,863 - INFO - 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec vm uve deleted from opserver 2025-08-28 04:06:34,863 - INFO - Validated that VM 59270a49-a54d-4a1f-bdfc-5c1b4b0d51ec is removed from Opserver 2025-08-28 04:06:34,863 - INFO - Deleting VM ctest-vn1_vm2-56544639 2025-08-28 04:06:34,930 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 04:06:34,930 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178 2025-08-28 04:06:34,938 - DEBUG - VM ID de9f8549-f809-4a1e-ad57-10a1afb79178 of VM ctest-vn1_vm2-56544639 is still found in API Server 2025-08-28 04:06:36,939 - DEBUG - Verifying in api server 10.0.0.41 2025-08-28 04:06:36,939 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178 2025-08-28 04:06:36,945 - DEBUG - Response Code: 404 2025-08-28 04:06:36,945 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178 2025-08-28 04:06:36,951 - DEBUG - Response Code: 404 2025-08-28 04:06:36,951 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/de9f8549-f809-4a1e-ad57-10a1afb79178 2025-08-28 04:06:36,956 - DEBUG - Response Code: 404 2025-08-28 04:06:36,956 - INFO - VM ctest-vn1_vm2-56544639 is fully removed in API-Server 2025-08-28 04:06:37,112 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found 2025-08-28 04:06:37,271 - DEBUG - VM route 192.168.1.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 of compute 10.0.0.65 2025-08-28 04:06:37,277 - DEBUG - Validated that vrouter 10.0.0.65 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 2025-08-28 04:06:37,277 - DEBUG - Validated that vrouter 10.0.0.27 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 2025-08-28 04:06:37,277 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 2025-08-28 04:06:37,277 - INFO - VM ctest-vn1_vm2-56544639 is removed in Compute, and routes are removed in all compute nodes 2025-08-28 04:06:37,493 - INFO - Routes for VM ctest-vn1_vm2-56544639 is removed in all control-nodes 2025-08-28 04:06:37,493 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:06:37,553 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:06:37,590 - DEBUG - VM de9f8549-f809-4a1e-ad57-10a1afb79178 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:06:37,590 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:06:37,590 - DEBUG - interface for vm de9f8549-f809-4a1e-ad57-10a1afb79178 deleted from vrouter uve ... 2025-08-28 04:06:37,590 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:06:40,591 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:06:40,648 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:06:40,675 - DEBUG - VM de9f8549-f809-4a1e-ad57-10a1afb79178 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:06:40,675 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:06:40,675 - DEBUG - interface for vm de9f8549-f809-4a1e-ad57-10a1afb79178 deleted from vrouter uve ... 2025-08-28 04:06:40,675 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:06:43,676 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:06:43,731 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:06:43,767 - DEBUG - VM de9f8549-f809-4a1e-ad57-10a1afb79178 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:06:43,767 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:06:43,767 - DEBUG - interface for vm de9f8549-f809-4a1e-ad57-10a1afb79178 deleted from vrouter uve ... 2025-08-28 04:06:43,767 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:06:46,768 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:06:46,822 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:06:46,850 - DEBUG - VM de9f8549-f809-4a1e-ad57-10a1afb79178 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:06:46,851 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:06:46,851 - DEBUG - interface for vm de9f8549-f809-4a1e-ad57-10a1afb79178 deleted from vrouter uve ... 2025-08-28 04:06:46,851 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:06:49,852 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:06:49,911 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:06:49,945 - DEBUG - VM de9f8549-f809-4a1e-ad57-10a1afb79178 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:06:49,945 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:06:49,945 - DEBUG - interface for vm de9f8549-f809-4a1e-ad57-10a1afb79178 deleted from vrouter uve ... 2025-08-28 04:06:49,945 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:06:52,945 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:06:53,037 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:06:53,073 - DEBUG - VM de9f8549-f809-4a1e-ad57-10a1afb79178 is still present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:06:53,073 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:06:53,073 - DEBUG - interface for vm de9f8549-f809-4a1e-ad57-10a1afb79178 deleted from vrouter uve ... 2025-08-28 04:06:53,073 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:06:56,074 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3956-1:Compute:contrail-vrouter-agent:0?flat 2025-08-28 04:06:56,134 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3956-1?flat 2025-08-28 04:06:56,172 - DEBUG - VM de9f8549-f809-4a1e-ad57-10a1afb79178 is not present in vrouter an-jenkins-deploy-platform-ansible-os-3956-1 uve 2025-08-28 04:06:56,172 - DEBUG - Verifying if the vm interface deleted from vroter uve... 2025-08-28 04:06:56,172 - DEBUG - interface for vm de9f8549-f809-4a1e-ad57-10a1afb79178 deleted from vrouter uve ... 2025-08-28 04:06:56,172 - DEBUG - Deleted interface not in error interface list ... 2025-08-28 04:06:56,172 - DEBUG - Validated that VM de9f8549-f809-4a1e-ad57-10a1afb79178 is removed in Vrouter UVE 2025-08-28 04:06:56,172 - DEBUG - Verifying through opserver in 10.0.0.19 2025-08-28 04:06:56,172 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577?flat 2025-08-28 04:06:56,197 - DEBUG - de9f8549-f809-4a1e-ad57-10a1afb79178 BM not in default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 uve 2025-08-28 04:06:56,197 - DEBUG - Validated that VM de9f8549-f809-4a1e-ad57-10a1afb79178 is not present in the VN UVE 2025-08-28 04:06:56,197 - DEBUG - Verifying the de9f8549-f809-4a1e-ad57-10a1afb79178 virtual network link through opserver 10.0.0.19 2025-08-28 04:06:56,197 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-machines 2025-08-28 04:06:56,213 - INFO - de9f8549-f809-4a1e-ad57-10a1afb79178 vm uve deleted from opserver 2025-08-28 04:06:56,213 - DEBUG - Verifying the de9f8549-f809-4a1e-ad57-10a1afb79178 virtual network link through opserver 10.0.0.38 2025-08-28 04:06:56,213 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-machines 2025-08-28 04:06:56,235 - INFO - de9f8549-f809-4a1e-ad57-10a1afb79178 vm uve deleted from opserver 2025-08-28 04:06:56,235 - DEBUG - Verifying the de9f8549-f809-4a1e-ad57-10a1afb79178 virtual network link through opserver 10.0.0.41 2025-08-28 04:06:56,235 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines 2025-08-28 04:06:56,252 - INFO - de9f8549-f809-4a1e-ad57-10a1afb79178 vm uve deleted from opserver 2025-08-28 04:06:56,252 - INFO - Validated that VM de9f8549-f809-4a1e-ad57-10a1afb79178 is removed from Opserver 2025-08-28 04:06:56,252 - INFO - Deleting VN ctest-vn3-22907884 2025-08-28 04:06:56,369 - INFO - Deleting VM ctest-vn2_vm1-38310879 2025-08-28 04:06:56,454 - INFO - Deleting VM ctest-vn1_vm1-24291330 2025-08-28 04:06:56,726 - INFO - Deleting VN ctest-vn2-66914093 2025-08-28 04:06:56,793 - DEBUG - RefsExistError Delete when resource still referred: ['http://10.0.0.41:8082/instance-ip/74ff2721-9b16-49a2-89d6-dd7db72c6ef6'] while deleting VN ctest-vn2-66914093..Will retry 2025-08-28 04:06:58,988 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/7fb41dd1-1da1-4362-9085-853e726280ec 2025-08-28 04:06:58,995 - DEBUG - Response Code: 404 2025-08-28 04:06:58,996 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 04:06:59,003 - DEBUG - Requesting: http://10.0.0.19:8082/domain/692c5a68-a199-4232-a07c-1cd4918de9d4 2025-08-28 04:06:59,029 - DEBUG - Requesting: http://10.0.0.19:8082/project/5631e639-3d56-4dbe-9639-2fd2c3fa5ac2 2025-08-28 04:06:59,132 - INFO - Validated that VN ctest-vn2-66914093 is not found in API Server 2025-08-28 04:06:59,143 - DEBUG - VN ctest-vn2-66914093 is not present in Agent 10.0.0.65 2025-08-28 04:06:59,152 - DEBUG - VN ctest-vn2-66914093 is not present in Agent 10.0.0.27 2025-08-28 04:06:59,152 - INFO - Validated that VN ctest-vn2-66914093 is not in any agent 2025-08-28 04:06:59,184 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-66914093 info 2025-08-28 04:06:59,332 - INFO - Deleting VN ctest-vn1-50579577 2025-08-28 04:06:59,465 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/36da7fbc-411a-4bba-ada9-874f8db9143c 2025-08-28 04:06:59,471 - DEBUG - Response Code: 404 2025-08-28 04:06:59,472 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-08-28 04:06:59,478 - DEBUG - Requesting: http://10.0.0.19:8082/domain/692c5a68-a199-4232-a07c-1cd4918de9d4 2025-08-28 04:06:59,505 - DEBUG - Requesting: http://10.0.0.19:8082/project/5631e639-3d56-4dbe-9639-2fd2c3fa5ac2 2025-08-28 04:06:59,599 - INFO - Validated that VN ctest-vn1-50579577 is not found in API Server 2025-08-28 04:06:59,610 - DEBUG - VN ctest-vn1-50579577 is not present in Agent 10.0.0.65 2025-08-28 04:06:59,621 - DEBUG - VN ctest-vn1-50579577 is not present in Agent 10.0.0.27 2025-08-28 04:06:59,621 - INFO - Validated that VN ctest-vn1-50579577 is not in any agent 2025-08-28 04:06:59,626 - DEBUG - VRF 2 is not seen in agent 10.0.0.65 2025-08-28 04:06:59,631 - DEBUG - Vrouter 10.0.0.65 does not have vrf 2 for VN ctest-vn1-50579577 2025-08-28 04:06:59,631 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestBasicVMVN0-72692947:ctest-vn1-50579577 2025-08-28 04:06:59,651 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-50579577 info 2025-08-28 04:06:59,690 - INFO - Deleted policy policy2 2025-08-28 04:06:59,725 - INFO - Deleted policy policy1 2025-08-28 04:07:00,989 - DEBUG - Skipping xmpp flap check 2025-08-28 04:07:00,990 - INFO - END TEST : test_process_restart_in_policy_between_vns : PASSED[0:05:33] 2025-08-28 04:07:00,990 - INFO - -------------------------------------------------------------------------------- 2025-08-28 04:07:00,993 - INFO - ================================================================================ 2025-08-28 04:07:00,993 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling 2025-08-28 04:07:00,993 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229). 1. Send broadcast traffic from one compute node. 2. Other compute in same subnet should receive that traffic. 3. Receiving compute should treat this traffic as underlay. 4. Compute should not replicate the packet and send the copy back. Pass criteria: Step 3-4 should pass Maintainer : chhandak@juniper.net 2025-08-28 04:07:02,278 - DEBUG - Skipping xmpp flap check 2025-08-28 04:07:02,278 - INFO - Initial checks done. Running the testcase now 2025-08-28 04:07:02,278 - INFO - 2025-08-28 04:07:02,656 - INFO - Agent interface name: ens6 2025-08-28 04:07:02,738 - DEBUG - Executing command: sudo tcpdump -nni ens6 -U host 0.0.0.0 -w /tmp/ens6_ctest-random-34396976.pcap 2025-08-28 04:07:08,059 - INFO - Pinging broacast address 0.0.0.0 from compute 10.20.0.193 2025-08-28 04:07:22,374 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}') 2025-08-28 04:07:24,512 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-34396976.pcap | grep -c length 2025-08-28 04:07:24,525 - DEBUG - STDOUT: 9 2025-08-28 04:07:24,525 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-34396976.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-28 04:07:24,525 - WARNING - 9 packets are found in tcpdump output file /tmp/ens6_ctest-random-34396976.pcap but expected 10 2025-08-28 04:07:26,525 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-34396976.pcap | grep -c length 2025-08-28 04:07:26,540 - DEBUG - STDOUT: 9 2025-08-28 04:07:26,540 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-34396976.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-28 04:07:26,540 - WARNING - 9 packets are found in tcpdump output file /tmp/ens6_ctest-random-34396976.pcap but expected 10 2025-08-28 04:07:28,541 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-34396976.pcap | grep -c length 2025-08-28 04:07:28,553 - DEBUG - STDOUT: 9 2025-08-28 04:07:28,553 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-34396976.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-28 04:07:28,553 - WARNING - 9 packets are found in tcpdump output file /tmp/ens6_ctest-random-34396976.pcap but expected 10 2025-08-28 04:07:30,554 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-34396976.pcap | grep -c length 2025-08-28 04:07:30,568 - DEBUG - STDOUT: 9 2025-08-28 04:07:30,568 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-34396976.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-28 04:07:30,568 - WARNING - 9 packets are found in tcpdump output file /tmp/ens6_ctest-random-34396976.pcap but expected 10 2025-08-28 04:07:32,569 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-34396976.pcap | grep -c length 2025-08-28 04:07:32,582 - DEBUG - STDOUT: 9 2025-08-28 04:07:32,582 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-34396976.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-28 04:07:32,582 - WARNING - 9 packets are found in tcpdump output file /tmp/ens6_ctest-random-34396976.pcap but expected 10 2025-08-28 04:07:34,583 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-34396976.pcap | grep -c length 2025-08-28 04:07:34,597 - DEBUG - STDOUT: 9 2025-08-28 04:07:34,597 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-34396976.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-28 04:07:34,597 - WARNING - 9 packets are found in tcpdump output file /tmp/ens6_ctest-random-34396976.pcap but expected 10 2025-08-28 04:07:36,598 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-34396976.pcap | grep -c length 2025-08-28 04:07:36,611 - DEBUG - STDOUT: 9 2025-08-28 04:07:36,611 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-34396976.pcap, link-type EN10MB (Ethernet), snapshot length 262144 2025-08-28 04:07:36,611 - WARNING - 9 packets are found in tcpdump output file /tmp/ens6_ctest-random-34396976.pcap but expected 10 2025-08-28 04:07:37,894 - ERROR - AssertionError Python 3.9.21: /usr/bin/python3 Thu Aug 28 04:07:36 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/serial_scripts/vm_regression/test_vm_serial.py in test_underlay_broadcast_traffic_handling(self=) 948 949 # Analyze pcap 950 assert verify_tcpdump_count(self, session, pcap, exp_count=packet_count), "There should only be %s\ 951 packet from source %s on compute %s" %(packet_count, broadcast_address, compute_ip) 952 self.logger.info( global verify_tcpdump_count = .deco_retry..f_retry> self = session = pcap = '/tmp/ens6_ctest-random-34396976.pcap' exp_count undefined packet_count = 10 broadcast_address = '0.0.0.0' compute_ip = '10.0.0.27' AssertionError: There should only be 10 packet from source 0.0.0.0 on compute 10.0.0.27 __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('There should only be 10 ... packet from source 0.0.0.0 on compute 10.0.0.27',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/serial_scripts/vm_regression/test_vm_serial.py", line 950, in test_underlay_broadcast_traffic_handling assert verify_tcpdump_count(self, session, pcap, exp_count=packet_count), "There should only be %s\ AssertionError: There should only be 10 packet from source 0.0.0.0 on compute 10.0.0.27 2025-08-28 04:07:37,896 - DEBUG - Skipping xmpp flap check 2025-08-28 04:07:37,896 - INFO - 2025-08-28 04:07:37,896 - INFO - END TEST : test_underlay_broadcast_traffic_handling : FAILED[0:00:37] 2025-08-28 04:07:37,896 - INFO - -------------------------------------------------------------------------------- 2025-08-28 04:07:38,677 - INFO - Deleted project: ctest-TestBasicVMVN0-72692947, ID : 5631e639-3d56-4dbe-9639-2fd2c3fa5ac2