2026-08-03 05:05:21,300 - INFO - Domain Default found not creating
2026-08-03 05:05:21,512 - INFO - Project ctest-TestBasicVMVN0-48746621 not found, creating it
2026-08-03 05:05:22,202 - INFO - Created Project:ctest-TestBasicVMVN0-48746621, ID : 34ffcb1a-09fd-4528-8d5b-a11d79da58f5
2026-08-03 05:05:24,103 - INFO - ================================================================================
2026-08-03 05:05:24,103 - INFO - STARTING TEST : test_control_node_switchover
2026-08-03 05:05:24,103 - INFO - TEST DESCRIPTION : Stop the control node and check peering with agent fallback to other control node.
1. Pick one VN from respource pool which has 2 VM's in it
2. Verify ping between VM's
3. Find active control node in cluster by agent inspect
4. Stop control service on active control node
5. Verify agents are connected to new active control-node using xmpp connections
6. Bring back control service on previous active node
7. Verify ping between VM's again after bringing up control serveice
Pass criteria: Step 2,5 and 7 should pass
2026-08-03 05:05:25,482 - DEBUG - Skipping xmpp flap check
2026-08-03 05:05:25,482 - INFO - Initial checks done. Running the testcase now
2026-08-03 05:05:25,482 - INFO -
2026-08-03 05:05:26,421 - DEBUG - Response for create_network : {'network': {'id': 'b661d1df-0f01-4878-a9d7-7d4211c600a2', 'name': 'ctest-vn1-49411142', 'tenant_id': '34ffcb1a09fd45288d5ba11d79da58f5', 'project_id': '34ffcb1a09fd45288d5ba11d79da58f5', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicVMVN0-48746621', 'ctest-vn1-49411142'], 'port_security_enabled': True, 'description': ''}}
2026-08-03 05:05:26,754 - DEBUG - Response for create_subnet : {'subnet': {'id': '02537349-0e28-4b4d-be84-dad08e002561', 'name': '', 'tenant_id': '34ffcb1a09fd45288d5ba11d79da58f5', 'network_id': 'b661d1df-0f01-4878-a9d7-7d4211c600a2', 'ip_version': 4, 'cidr': '192.168.1.0/24', 'allocation_pools': [{'start': '192.168.1.2', 'end': '192.168.1.254'}], 'gateway_ip': '192.168.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '192.168.1.2', 'tags': [], 'project_id': '34ffcb1a09fd45288d5ba11d79da58f5'}}
2026-08-03 05:05:26,782 - INFO - Created VN ctest-vn1-49411142
2026-08-03 05:05:26,801 - DEBUG - VN ctest-vn1-49411142 UUID is b661d1df-0f01-4878-a9d7-7d4211c600a2
2026-08-03 05:05:26,891 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2026-08-03 05:05:27,277 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2026-08-03 05:05:27,308 - DEBUG - Requesting: http://10.0.0.249:8082/domain/724de60e-9f87-4d23-867c-6bd9a9880f5b
2026-08-03 05:05:27,327 - DEBUG - Requesting: http://10.0.0.249:8082/project/34ffcb1a-09fd-4528-8d5b-a11d79da58f5
2026-08-03 05:05:27,463 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/b661d1df-0f01-4878-a9d7-7d4211c600a2
2026-08-03 05:05:27,478 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/b661d1df-0f01-4878-a9d7-7d4211c600a2
2026-08-03 05:05:27,494 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/63451959-1b72-4828-9be7-8a838a487dab
2026-08-03 05:05:27,509 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/63451959-1b72-4828-9be7-8a838a487dab
2026-08-03 05:05:27,524 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/2b05ffe8-9834-4853-b1f0-91c1c39f254a
2026-08-03 05:05:27,536 - DEBUG - Route Targets: ['target:64512:8000004']
2026-08-03 05:05:27,536 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/b661d1df-0f01-4878-a9d7-7d4211c600a2
2026-08-03 05:05:27,551 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/63451959-1b72-4828-9be7-8a838a487dab
2026-08-03 05:05:27,574 - INFO - Verified VN network id 13 for VN b661d1df-0f01-4878-a9d7-7d4211c600a2
2026-08-03 05:05:27,575 - INFO - Verifications in API Server for VN ctest-vn1-49411142 passed
2026-08-03 05:05:27,575 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/b661d1df-0f01-4878-a9d7-7d4211c600a2
2026-08-03 05:05:27,588 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/63451959-1b72-4828-9be7-8a838a487dab
2026-08-03 05:05:27,599 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/63451959-1b72-4828-9be7-8a838a487dab
2026-08-03 05:05:27,612 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/2b05ffe8-9834-4853-b1f0-91c1c39f254a
2026-08-03 05:05:27,629 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-48746621', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13142015943603472504', 'uuid-lslong': '12238388235122114722'}, 'enable': 'true', 'created': '2026-08-03T05:05:26', 'last-modified': '2026-08-03T05:05:26', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.945016'}
2026-08-03 05:05:27,637 - DEBUG - Route Targets: ['target:64512:8000004']
2026-08-03 05:05:27,643 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-48746621', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13142015943603472504', 'uuid-lslong': '12238388235122114722'}, 'enable': 'true', 'created': '2026-08-03T05:05:26', 'last-modified': '2026-08-03T05:05:26', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.957515'}
2026-08-03 05:05:27,650 - DEBUG - Route Targets: ['target:64512:8000004']
2026-08-03 05:05:27,658 - DEBUG - Control-node 10.0.0.161 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-48746621', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '13142015943603472504', 'uuid-lslong': '12238388235122114722'}, 'enable': 'true', 'created': '2026-08-03T05:05:26', 'last-modified': '2026-08-03T05:05:26', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.970849'}
2026-08-03 05:05:27,667 - DEBUG - Route Targets: ['target:64512:8000004']
2026-08-03 05:05:27,667 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-49411142 passed
2026-08-03 05:05:27,668 - DEBUG - ====Verifying policy data for ctest-vn1-49411142 in API_Server ======
2026-08-03 05:05:27,668 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2026-08-03 05:05:27,676 - DEBUG - Requesting: http://10.0.0.249:8082/domain/724de60e-9f87-4d23-867c-6bd9a9880f5b
2026-08-03 05:05:27,690 - DEBUG - Requesting: http://10.0.0.249:8082/project/34ffcb1a-09fd-4528-8d5b-a11d79da58f5
2026-08-03 05:05:27,799 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/b661d1df-0f01-4878-a9d7-7d4211c600a2
2026-08-03 05:05:27,813 - DEBUG - =>VN ctest-vn1-49411142 has no policy to be verified
2026-08-03 05:05:27,813 - DEBUG - Verifying the vn in opserver
2026-08-03 05:05:27,813 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 virtual network link through opserver 10.0.0.249
2026-08-03 05:05:27,813 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks
2026-08-03 05:05:28,191 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks
2026-08-03 05:05:28,258 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142?flat'}
2026-08-03 05:05:28,258 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 virtual network link through opserver 10.0.0.50
2026-08-03 05:05:28,258 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks
2026-08-03 05:05:28,647 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks
2026-08-03 05:05:28,714 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142?flat'}
2026-08-03 05:05:28,714 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 virtual network link through opserver 10.0.0.161
2026-08-03 05:05:28,714 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-networks
2026-08-03 05:05:29,099 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-networks
2026-08-03 05:05:29,167 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142', 'href': 'http://10.0.0.161:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142?flat'}
2026-08-03 05:05:29,167 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is found in opserver
2026-08-03 05:05:29,194 - DEBUG - Do not have enough data to verify VN in agent
2026-08-03 05:05:29,209 - DEBUG - VRF ids for VN ctest-vn1-49411142: {}
2026-08-03 05:05:29,275 - DEBUG - Services list from nova: [, , , ]
2026-08-03 05:05:29,784 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-6129-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-6129-2)
2026-08-03 05:05:30,440 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-6129-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-6129-1)
2026-08-03 05:05:30,440 - INFO - Waiting for VM ctest-vn1_vm1-80444148 to be up..
2026-08-03 05:05:30,506 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:05:35,639 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:05:40,760 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:05:45,869 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:05:51,004 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:05:56,117 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:06:01,234 - DEBUG - VM is in ACTIVE state now
2026-08-03 05:06:01,234 - INFO - VM name : ctest-vn1_vm1-80444148
2026-08-03 05:06:01,347 - DEBUG - VM ctest-vn1_vm1-80444148 ID is f7ecf15c-6909-44e8-98a2-19d1ead5fbce
2026-08-03 05:06:01,380 - DEBUG - VM ctest-vn1_vm1-80444148 launched on Node an-jenkins-deploy-platform-ansible-os-6129-2
2026-08-03 05:06:01,554 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce
2026-08-03 05:06:01,950 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce
2026-08-03 05:06:01,998 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/d8d19501-3ccb-41af-819d-365ecd5ed8e2
2026-08-03 05:06:05,222 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms')
2026-08-03 05:06:05,222 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-80444148 failed!
2026-08-03 05:06:05,242 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is 192.168.1.1 and allocation pool is NOT set
2026-08-03 05:06:09,321 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms')
2026-08-03 05:06:09,321 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-80444148 failed!
2026-08-03 05:06:09,341 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is 192.168.1.1 and allocation pool is NOT set
2026-08-03 05:06:13,446 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1037ms')
2026-08-03 05:06:13,446 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-80444148 failed!
2026-08-03 05:06:13,467 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is 192.168.1.1 and allocation pool is NOT set
2026-08-03 05:06:17,578 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1032ms')
2026-08-03 05:06:17,578 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-80444148 failed!
2026-08-03 05:06:17,598 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is 192.168.1.1 and allocation pool is NOT set
2026-08-03 05:06:21,706 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1031ms')
2026-08-03 05:06:21,707 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-80444148 failed!
2026-08-03 05:06:21,728 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is 192.168.1.1 and allocation pool is NOT set
2026-08-03 05:06:25,825 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1034ms')
2026-08-03 05:06:25,825 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-80444148 failed!
2026-08-03 05:06:25,845 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is 192.168.1.1 and allocation pool is NOT set
2026-08-03 05:06:29,932 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms')
2026-08-03 05:06:29,932 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-80444148 failed!
2026-08-03 05:06:29,955 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is 192.168.1.1 and allocation pool is NOT set
2026-08-03 05:06:34,050 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1025ms')
2026-08-03 05:06:34,050 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-80444148 failed!
2026-08-03 05:06:34,071 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is 192.168.1.1 and allocation pool is NOT set
2026-08-03 05:06:36,145 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=13.8 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=2.86 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 2.855/8.345/13.835/5.490 ms')
2026-08-03 05:06:36,145 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-80444148 passed
2026-08-03 05:06:36,309 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:06:36,309 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-80444148, IP 192.168.1.3, Port 22
2026-08-03 05:06:36,378 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2026-08-03 05:06:36,471 - DEBUG - VM ctest-vn1_vm1-80444148 is NOT ready for SSH connections, VM status: ACTIVE
2026-08-03 05:06:41,472 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:06:41,472 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-80444148, IP 192.168.1.3, Port 22
2026-08-03 05:06:41,542 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2026-08-03 05:06:41,677 - DEBUG - VM ctest-vn1_vm1-80444148 is NOT ready for SSH connections, VM status: ACTIVE
2026-08-03 05:06:46,678 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:06:46,678 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-80444148, IP 192.168.1.3, Port 22
2026-08-03 05:06:46,745 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2026-08-03 05:06:46,846 - DEBUG - VM ctest-vn1_vm1-80444148 is NOT ready for SSH connections, VM status: ACTIVE
2026-08-03 05:06:51,846 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:06:51,847 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-80444148, IP 192.168.1.3, Port 22
2026-08-03 05:06:52,022 - DEBUG - VM ctest-vn1_vm1-80444148 is ready for SSH connections
2026-08-03 05:06:52,022 - INFO - Waiting for VM ctest-vn1_vm2-26521843 to be up..
2026-08-03 05:06:52,151 - DEBUG - VM is in ACTIVE state now
2026-08-03 05:06:52,151 - INFO - VM name : ctest-vn1_vm2-26521843
2026-08-03 05:06:52,267 - DEBUG - VM ctest-vn1_vm2-26521843 ID is 7541dfc5-c4be-4082-b6ad-ab2e70520fd9
2026-08-03 05:06:52,267 - DEBUG - VM ctest-vn1_vm2-26521843 launched on Node an-jenkins-deploy-platform-ansible-os-6129-1
2026-08-03 05:06:52,388 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9
2026-08-03 05:06:52,404 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/06e93982-f5ea-4ddf-82b8-dd1ed9b4fb4a
2026-08-03 05:06:53,605 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=11.9 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=3.80 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 3.800/7.828/11.857/4.028 ms')
2026-08-03 05:06:53,605 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-26521843 passed
2026-08-03 05:06:53,695 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:06:53,695 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-26521843, IP 192.168.1.4, Port 22
2026-08-03 05:06:53,765 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2026-08-03 05:06:53,890 - DEBUG - VM ctest-vn1_vm2-26521843 is NOT ready for SSH connections, VM status: ACTIVE
2026-08-03 05:06:58,891 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:06:58,891 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-26521843, IP 192.168.1.4, Port 22
2026-08-03 05:06:58,971 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2026-08-03 05:06:59,069 - DEBUG - VM ctest-vn1_vm2-26521843 is NOT ready for SSH connections, VM status: ACTIVE
2026-08-03 05:07:04,069 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:07:04,069 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-26521843, IP 192.168.1.4, Port 22
2026-08-03 05:07:04,245 - DEBUG - VM ctest-vn1_vm2-26521843 is ready for SSH connections
2026-08-03 05:07:04,246 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:07:04,246 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123
2026-08-03 05:07:04,246 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4
2026-08-03 05:07:58,406 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=16.3 ms
64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=2.16 ms
64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.91 ms
--- 192.168.1.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.916/6.805/16.333/6.738 ms
2026-08-03 05:07:58,406 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-80444148 passed
2026-08-03 05:07:58,406 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:07:58,406 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.38, gateway password: c0ntrail123
2026-08-03 05:07:58,407 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3
2026-08-03 05:08:52,496 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=15.0 ms
64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.70 ms
64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.78 ms
--- 192.168.1.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.704/6.163/15.003/6.251 ms
2026-08-03 05:08:52,497 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-26521843 passed
2026-08-03 05:08:52,505 - INFO - Active control node from the Agent 10.0.0.19 is 10.0.0.161
2026-08-03 05:08:52,506 - INFO - Stoping the Control service in 10.0.0.161
2026-08-03 05:08:53,047 - INFO - 10.0.0.161
2026-08-03 05:08:53,047 - INFO - control:inactive
2026-08-03 05:08:53,047 - INFO - Contrail services control are down on nodes 10.0.0.161
2026-08-03 05:08:58,056 - INFO - Active control node from the Agent 10.0.0.19 is 10.0.0.249
2026-08-03 05:08:58,056 - INFO - Starting the Control service in 10.0.0.161
2026-08-03 05:08:58,491 - INFO - 10.0.0.161
2026-08-03 05:08:58,497 - INFO - control:initializing
2026-08-03 05:08:58,497 - DEBUG - defaultdict(, {'10.0.0.161': {'control': {'status': 'initializing', 'description': None}}})
2026-08-03 05:08:58,497 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2026-08-03 05:09:03,620 - INFO - 10.0.0.161
2026-08-03 05:09:03,627 - INFO - control:active
2026-08-03 05:09:03,627 - INFO - Contrail services ['control'] are up on nodes 10.0.0.161
2026-08-03 05:09:08,756 - DEBUG - VM is in ACTIVE state now
2026-08-03 05:09:08,756 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:09:08,756 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce
2026-08-03 05:09:08,768 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/d8d19501-3ccb-41af-819d-365ecd5ed8e2
2026-08-03 05:09:08,784 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce
2026-08-03 05:09:08,797 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/d8d19501-3ccb-41af-819d-365ecd5ed8e2
2026-08-03 05:09:08,815 - DEBUG - Requesting: http://10.0.0.161:8082/instance-ip/6c960fda-5619-43ab-9e98-bb0f464135ae
2026-08-03 05:09:08,831 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:09:08,831 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce
2026-08-03 05:09:08,842 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/d8d19501-3ccb-41af-819d-365ecd5ed8e2
2026-08-03 05:09:08,855 - DEBUG - Requesting: http://10.0.0.161:8082/instance-ip/6c960fda-5619-43ab-9e98-bb0f464135ae
2026-08-03 05:09:08,868 - INFO - VM ctest-vn1_vm1-80444148 verfication in all API Servers passed
2026-08-03 05:09:08,896 - DEBUG - VM ctest-vn1_vm1-80444148 Tap interface: {'index': '3', 'name': 'tapd8d19501-3c', 'uuid': 'd8d19501-3ccb-41af-819d-365ecd5ed8e2', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142:ctest-vn1-49411142', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142', 'vm_uuid': 'f7ecf15c-6909-44e8-98a2-19d1ead5fbce', 'vm_name': 'ctest-vn1_vm1-80444148', 'ip_addr': '192.168.1.3', 'mac_addr': '02:d8:d1:95:01:3c', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '141', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-48746621:d8d19501-3ccb-41af-819d-365ecd5ed8e2', 'sg_uuid_list': ['4360d165-c92f-4ce9-887c-ba23a9f642bf'], 'static_route_list': None, 'vm_project_uuid': '34ffcb1a-09fd-4528-8d5b-a11d79da58f5', 'admin_state': 'Enabled', 'flow_key_idx': '21', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2026-08-03 05:09:08,896 - DEBUG - Agent 10.0.0.19 vrf name: default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142:ctest-vn1-49411142
2026-08-03 05:09:08,904 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142:ctest-vn1-49411142', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.254:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]}
2026-08-03 05:09:08,999 - DEBUG - VM ctest-vn1_vm1-80444148 : Tap interface tapd8d19501-3c is set to Active
2026-08-03 05:09:09,000 - DEBUG - Tap interface tapd8d19501-3c detail : {'index': '3', 'name': 'tapd8d19501-3c', 'uuid': 'd8d19501-3ccb-41af-819d-365ecd5ed8e2', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142:ctest-vn1-49411142', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142', 'vm_uuid': 'f7ecf15c-6909-44e8-98a2-19d1ead5fbce', 'vm_name': 'ctest-vn1_vm1-80444148', 'ip_addr': '192.168.1.3', 'mac_addr': '02:d8:d1:95:01:3c', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '141', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-48746621:d8d19501-3ccb-41af-819d-365ecd5ed8e2', 'sg_uuid_list': ['4360d165-c92f-4ce9-887c-ba23a9f642bf'], 'static_route_list': None, 'vm_project_uuid': '34ffcb1a-09fd-4528-8d5b-a11d79da58f5', 'admin_state': 'Enabled', 'flow_key_idx': '21', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2026-08-03 05:09:09,127 - DEBUG - Starting Layer 2 verification in Agent
2026-08-03 05:09:09,143 - DEBUG - Layer 2 path is seen for VM MAC 02:d8:d1:95:01:3c in agent 10.0.0.19
2026-08-03 05:09:09,143 - DEBUG - Active layer 2 route in agent is present for VMI tapd8d19501-3c
2026-08-03 05:09:09,143 - DEBUG - L2 label(29) matches bw route table and interface table
2026-08-03 05:09:09,150 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is found in Agent of node 10.0.0.38
2026-08-03 05:09:09,291 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is consistent in agent 10.0.0.38
2026-08-03 05:09:09,292 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.38
2026-08-03 05:09:09,292 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 verification for VM ctest-vn1_vm1-80444148 in Agent 10.0.0.38 passed
2026-08-03 05:09:09,414 - DEBUG - Starting all layer 2 verification in agent 10.0.0.38
2026-08-03 05:09:09,428 - DEBUG - Route for VM MAC 02:d8:d1:95:01:3c is consistent in agent 10.0.0.38
2026-08-03 05:09:09,436 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is found in Agent of node 10.0.0.19
2026-08-03 05:09:09,585 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is consistent in agent 10.0.0.19
2026-08-03 05:09:09,586 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.19
2026-08-03 05:09:09,586 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 verification for VM ctest-vn1_vm1-80444148 in Agent 10.0.0.19 passed
2026-08-03 05:09:09,706 - DEBUG - Starting all layer 2 verification in agent 10.0.0.19
2026-08-03 05:09:09,723 - DEBUG - Route for VM MAC 02:d8:d1:95:01:3c is consistent in agent 10.0.0.19
2026-08-03 05:09:09,723 - INFO - VM ctest-vn1_vm1-80444148 verifications in Compute nodes passed
2026-08-03 05:09:10,041 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.38
2026-08-03 05:09:10,047 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.254, 24; Got 10.20.0.254,24
2026-08-03 05:09:10,047 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.38
2026-08-03 05:09:10,198 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.19
2026-08-03 05:09:10,204 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0
2026-08-03 05:09:10,204 - INFO - Validated routes of VM ctest-vn1_vm1-80444148 in all vrouters
2026-08-03 05:09:10,791 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node
2026-08-03 05:09:10,891 - DEBUG - Layer2 route found for VM MAC 02:d8:d1:95:01:3c in Control-node 10.0.0.50
2026-08-03 05:09:10,891 - DEBUG - L2 Label for VM ctest-vn1_vm1-80444148 same between Control-node 10.0.0.50 and Agent, Expected: 29, Seen: 29
2026-08-03 05:09:10,891 - DEBUG - Starting all layer2 verification in 10.0.0.161 Control Node
2026-08-03 05:09:11,046 - DEBUG - Layer2 route found for VM MAC 02:d8:d1:95:01:3c in Control-node 10.0.0.161
2026-08-03 05:09:11,046 - DEBUG - L2 Label for VM ctest-vn1_vm1-80444148 same between Control-node 10.0.0.161 and Agent, Expected: 29, Seen: 29
2026-08-03 05:09:11,046 - DEBUG - Starting all layer2 verification in 10.0.0.249 Control Node
2026-08-03 05:09:11,215 - DEBUG - Layer2 route found for VM MAC 02:d8:d1:95:01:3c in Control-node 10.0.0.249
2026-08-03 05:09:11,215 - DEBUG - L2 Label for VM ctest-vn1_vm1-80444148 same between Control-node 10.0.0.249 and Agent, Expected: 29, Seen: 29
2026-08-03 05:09:11,215 - INFO - Verification in Control-nodes for VM ctest-vn1_vm1-80444148 passed
2026-08-03 05:09:11,215 - DEBUG - Verifying the vm in opserver
2026-08-03 05:09:11,215 - DEBUG - Verifying in collector 10.0.0.161 ...
2026-08-03 05:09:11,215 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce?flat
2026-08-03 05:09:11,298 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-48746621:d8d19501-3ccb-41af-819d-365ecd5ed8e2?flat
2026-08-03 05:09:11,326 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-48746621:d8d19501-3ccb-41af-819d-365ecd5ed8e2?flat
2026-08-03 05:09:11,404 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce?flat
2026-08-03 05:09:11,476 - DEBUG - Verifying vm in vn uve
2026-08-03 05:09:11,476 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-48746621:d8d19501-3ccb-41af-819d-365ecd5ed8e2?flat
2026-08-03 05:09:11,557 - DEBUG - VM uve shows interface as {'ip6_address': '::', 'fixed_ip4_list': ['192.168.1.3'], 'ip6_active': False, 'active': True, 'mac_address': '02:d8:d1:95:01:3c', 'port_mirror_enabled': False, 'rx_vlan': 65535, 'uuid': 'd8d19501-3ccb-41af-819d-365ecd5ed8e2', 'vm_name': 'ctest-vn1_vm1-80444148', 'ip4_active': True, 'admin_state': True, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142', 'vm_uuid': 'f7ecf15c-6909-44e8-98a2-19d1ead5fbce', 'is_health_check_active': True, 'ip_address': '192.168.1.3', 'vn_uuid': 'b661d1df-0f01-4878-a9d7-7d4211c600a2', 'tx_vlan': 65535, 'gateway': '192.168.1.1', 'label': 24, 'l2_active': True, 'vhostuser_mode': 0, '__T': 1785733751535342}
2026-08-03 05:09:11,557 - DEBUG - VM uve shows ip address as ['192.168.1.3', '::']
2026-08-03 05:09:11,557 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142
2026-08-03 05:09:11,557 - INFO - Verifying through opserver in 10.0.0.249
2026-08-03 05:09:11,557 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142?flat
2026-08-03 05:09:11,608 - DEBUG - expected vm list f7ecf15c-6909-44e8-98a2-19d1ead5fbce
2026-08-03 05:09:11,608 - DEBUG - Extracted vm list ['f7ecf15c-6909-44e8-98a2-19d1ead5fbce']
2026-08-03 05:09:11,608 - INFO - VM f7ecf15c-6909-44e8-98a2-19d1ead5fbce is present in default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142
2026-08-03 05:09:11,608 - DEBUG - Verifying vm in vrouter uve
2026-08-03 05:09:11,608 - DEBUG - Getting info from collector 10.0.0.161..
2026-08-03 05:09:11,608 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce?flat
2026-08-03 05:09:11,632 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-6129-2']
2026-08-03 05:09:11,632 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-2:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:09:11,676 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-2?flat
2026-08-03 05:09:11,712 - DEBUG - VM f7ecf15c-6909-44e8-98a2-19d1ead5fbce is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-2 uve
2026-08-03 05:09:11,712 - DEBUG - Validated that VM ctest-vn1_vm1-80444148 is in Vrouter an-jenkins-deploy-platform-ansible-os-6129-2 UVE
2026-08-03 05:09:11,712 - DEBUG - Verifying vm tap interface/vn in vrouter uve
2026-08-03 05:09:11,723 - DEBUG - Expected tap interface of VM uuid f7ecf15c-6909-44e8-98a2-19d1ead5fbce is default-domain:ctest-TestBasicVMVN0-48746621:d8d19501-3ccb-41af-819d-365ecd5ed8e2
2026-08-03 05:09:11,723 - DEBUG - Expected VN of VM uuid f7ecf15c-6909-44e8-98a2-19d1ead5fbce is default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142
2026-08-03 05:09:11,723 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-2:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:09:11,765 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-2?flat
2026-08-03 05:09:11,861 - DEBUG - VM f7ecf15c-6909-44e8-98a2-19d1ead5fbce is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-2 uve
2026-08-03 05:09:11,861 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-48746621:d8d19501-3ccb-41af-819d-365ecd5ed8e2 of vm f7ecf15c-6909-44e8-98a2-19d1ead5fbce is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-2 uve
2026-08-03 05:09:11,862 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 of vm f7ecf15c-6909-44e8-98a2-19d1ead5fbce is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-2 uve
2026-08-03 05:09:11,862 - DEBUG - Verifying the f7ecf15c-6909-44e8-98a2-19d1ead5fbce virtual network link through opserver 10.0.0.249
2026-08-03 05:09:11,862 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machines
2026-08-03 05:09:11,880 - DEBUG - VM link and name as {'name': 'f7ecf15c-6909-44e8-98a2-19d1ead5fbce', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce?flat'}
2026-08-03 05:09:11,880 - DEBUG - Verifying the f7ecf15c-6909-44e8-98a2-19d1ead5fbce virtual network link through opserver 10.0.0.50
2026-08-03 05:09:11,880 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines
2026-08-03 05:09:11,955 - DEBUG - VM link and name as {'name': 'f7ecf15c-6909-44e8-98a2-19d1ead5fbce', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce?flat'}
2026-08-03 05:09:11,955 - DEBUG - Verifying the f7ecf15c-6909-44e8-98a2-19d1ead5fbce virtual network link through opserver 10.0.0.161
2026-08-03 05:09:11,955 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machines
2026-08-03 05:09:11,978 - DEBUG - VM link and name as {'name': 'f7ecf15c-6909-44e8-98a2-19d1ead5fbce', 'href': 'http://10.0.0.161:8081/analytics/uves/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce?flat'}
2026-08-03 05:09:11,978 - INFO - VM ctest-vn1_vm1-80444148 validations in Opserver passed
2026-08-03 05:09:12,103 - DEBUG - VM is in ACTIVE state now
2026-08-03 05:09:12,103 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:09:12,103 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9
2026-08-03 05:09:12,115 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/06e93982-f5ea-4ddf-82b8-dd1ed9b4fb4a
2026-08-03 05:09:12,136 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9
2026-08-03 05:09:12,152 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/06e93982-f5ea-4ddf-82b8-dd1ed9b4fb4a
2026-08-03 05:09:12,172 - DEBUG - Requesting: http://10.0.0.161:8082/instance-ip/cfea4736-913a-4f43-a7aa-e15e0c322f96
2026-08-03 05:09:12,188 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:09:12,189 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9
2026-08-03 05:09:12,205 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/06e93982-f5ea-4ddf-82b8-dd1ed9b4fb4a
2026-08-03 05:09:12,222 - DEBUG - Requesting: http://10.0.0.161:8082/instance-ip/cfea4736-913a-4f43-a7aa-e15e0c322f96
2026-08-03 05:09:12,236 - INFO - VM ctest-vn1_vm2-26521843 verfication in all API Servers passed
2026-08-03 05:09:12,258 - DEBUG - VM ctest-vn1_vm2-26521843 Tap interface: {'index': '3', 'name': 'tap06e93982-f5', 'uuid': '06e93982-f5ea-4ddf-82b8-dd1ed9b4fb4a', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142:ctest-vn1-49411142', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142', 'vm_uuid': '7541dfc5-c4be-4082-b6ad-ab2e70520fd9', 'vm_name': 'ctest-vn1_vm2-26521843', 'ip_addr': '192.168.1.4', 'mac_addr': '02:06:e9:39:82:f5', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '97', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-48746621:06e93982-f5ea-4ddf-82b8-dd1ed9b4fb4a', 'sg_uuid_list': ['4360d165-c92f-4ce9-887c-ba23a9f642bf'], 'static_route_list': None, 'vm_project_uuid': '34ffcb1a-09fd-4528-8d5b-a11d79da58f5', 'admin_state': 'Enabled', 'flow_key_idx': '22', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2026-08-03 05:09:12,258 - DEBUG - Agent 10.0.0.38 vrf name: default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142:ctest-vn1-49411142
2026-08-03 05:09:12,264 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142:ctest-vn1-49411142', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.31:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]}
2026-08-03 05:09:12,363 - DEBUG - VM ctest-vn1_vm2-26521843 : Tap interface tap06e93982-f5 is set to Active
2026-08-03 05:09:12,363 - DEBUG - Tap interface tap06e93982-f5 detail : {'index': '3', 'name': 'tap06e93982-f5', 'uuid': '06e93982-f5ea-4ddf-82b8-dd1ed9b4fb4a', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142:ctest-vn1-49411142', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '25', 'l2_label': '29', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142', 'vm_uuid': '7541dfc5-c4be-4082-b6ad-ab2e70520fd9', 'vm_name': 'ctest-vn1_vm2-26521843', 'ip_addr': '192.168.1.4', 'mac_addr': '02:06:e9:39:82:f5', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '97', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-48746621:06e93982-f5ea-4ddf-82b8-dd1ed9b4fb4a', 'sg_uuid_list': ['4360d165-c92f-4ce9-887c-ba23a9f642bf'], 'static_route_list': None, 'vm_project_uuid': '34ffcb1a-09fd-4528-8d5b-a11d79da58f5', 'admin_state': 'Enabled', 'flow_key_idx': '22', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2026-08-03 05:09:12,484 - DEBUG - Starting Layer 2 verification in Agent
2026-08-03 05:09:12,502 - DEBUG - Layer 2 path is seen for VM MAC 02:06:e9:39:82:f5 in agent 10.0.0.38
2026-08-03 05:09:12,503 - DEBUG - Active layer 2 route in agent is present for VMI tap06e93982-f5
2026-08-03 05:09:12,503 - DEBUG - L2 label(29) matches bw route table and interface table
2026-08-03 05:09:12,512 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is found in Agent of node 10.0.0.38
2026-08-03 05:09:12,661 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is consistent in agent 10.0.0.38
2026-08-03 05:09:12,661 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.38
2026-08-03 05:09:12,661 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 verification for VM ctest-vn1_vm2-26521843 in Agent 10.0.0.38 passed
2026-08-03 05:09:12,796 - DEBUG - Starting all layer 2 verification in agent 10.0.0.38
2026-08-03 05:09:12,823 - DEBUG - Route for VM MAC 02:06:e9:39:82:f5 is consistent in agent 10.0.0.38
2026-08-03 05:09:12,837 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is found in Agent of node 10.0.0.19
2026-08-03 05:09:12,944 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 is consistent in agent 10.0.0.19
2026-08-03 05:09:12,944 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.19
2026-08-03 05:09:12,944 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 verification for VM ctest-vn1_vm2-26521843 in Agent 10.0.0.19 passed
2026-08-03 05:09:13,075 - DEBUG - Starting all layer 2 verification in agent 10.0.0.19
2026-08-03 05:09:13,087 - DEBUG - Route for VM MAC 02:06:e9:39:82:f5 is consistent in agent 10.0.0.19
2026-08-03 05:09:13,087 - INFO - VM ctest-vn1_vm2-26521843 verifications in Compute nodes passed
2026-08-03 05:09:13,429 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.38
2026-08-03 05:09:13,443 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0
2026-08-03 05:09:13,618 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.19
2026-08-03 05:09:13,625 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.31, 25; Got 10.20.0.31,25
2026-08-03 05:09:13,625 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.19
2026-08-03 05:09:13,625 - INFO - Validated routes of VM ctest-vn1_vm2-26521843 in all vrouters
2026-08-03 05:09:14,240 - DEBUG - Starting all layer2 verification in 10.0.0.50 Control Node
2026-08-03 05:09:14,410 - DEBUG - Layer2 route found for VM MAC 02:06:e9:39:82:f5 in Control-node 10.0.0.50
2026-08-03 05:09:14,410 - DEBUG - L2 Label for VM ctest-vn1_vm2-26521843 same between Control-node 10.0.0.50 and Agent, Expected: 29, Seen: 29
2026-08-03 05:09:14,410 - DEBUG - Starting all layer2 verification in 10.0.0.161 Control Node
2026-08-03 05:09:14,565 - DEBUG - Layer2 route found for VM MAC 02:06:e9:39:82:f5 in Control-node 10.0.0.161
2026-08-03 05:09:14,566 - DEBUG - L2 Label for VM ctest-vn1_vm2-26521843 same between Control-node 10.0.0.161 and Agent, Expected: 29, Seen: 29
2026-08-03 05:09:14,566 - DEBUG - Starting all layer2 verification in 10.0.0.249 Control Node
2026-08-03 05:09:14,727 - DEBUG - Layer2 route found for VM MAC 02:06:e9:39:82:f5 in Control-node 10.0.0.249
2026-08-03 05:09:14,730 - DEBUG - L2 Label for VM ctest-vn1_vm2-26521843 same between Control-node 10.0.0.249 and Agent, Expected: 29, Seen: 29
2026-08-03 05:09:14,730 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-26521843 passed
2026-08-03 05:09:14,730 - DEBUG - Verifying the vm in opserver
2026-08-03 05:09:14,730 - DEBUG - Verifying in collector 10.0.0.161 ...
2026-08-03 05:09:14,730 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9?flat
2026-08-03 05:09:14,757 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-48746621:06e93982-f5ea-4ddf-82b8-dd1ed9b4fb4a?flat
2026-08-03 05:09:14,789 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-48746621:06e93982-f5ea-4ddf-82b8-dd1ed9b4fb4a?flat
2026-08-03 05:09:14,820 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9?flat
2026-08-03 05:09:14,849 - DEBUG - Verifying vm in vn uve
2026-08-03 05:09:14,849 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-48746621:06e93982-f5ea-4ddf-82b8-dd1ed9b4fb4a?flat
2026-08-03 05:09:14,880 - DEBUG - VM uve shows interface as {'ip6_address': '::', 'fixed_ip4_list': ['192.168.1.4'], 'ip6_active': False, 'active': True, 'mac_address': '02:06:e9:39:82:f5', 'port_mirror_enabled': False, 'rx_vlan': 65535, 'uuid': '06e93982-f5ea-4ddf-82b8-dd1ed9b4fb4a', 'vm_name': 'ctest-vn1_vm2-26521843', 'ip4_active': True, 'admin_state': True, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142', 'vm_uuid': '7541dfc5-c4be-4082-b6ad-ab2e70520fd9', 'is_health_check_active': True, 'ip_address': '192.168.1.4', 'vn_uuid': 'b661d1df-0f01-4878-a9d7-7d4211c600a2', 'tx_vlan': 65535, 'gateway': '192.168.1.1', 'label': 25, 'l2_active': True, 'vhostuser_mode': 0, '__T': 1785733563178343}
2026-08-03 05:09:14,880 - DEBUG - VM uve shows ip address as ['192.168.1.4', '::']
2026-08-03 05:09:14,880 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142
2026-08-03 05:09:14,880 - INFO - Verifying through opserver in 10.0.0.249
2026-08-03 05:09:14,880 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142?flat
2026-08-03 05:09:14,921 - DEBUG - expected vm list 7541dfc5-c4be-4082-b6ad-ab2e70520fd9
2026-08-03 05:09:14,921 - DEBUG - Extracted vm list ['7541dfc5-c4be-4082-b6ad-ab2e70520fd9']
2026-08-03 05:09:14,921 - INFO - VM 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 is present in default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142
2026-08-03 05:09:14,922 - DEBUG - Verifying vm in vrouter uve
2026-08-03 05:09:14,922 - DEBUG - Getting info from collector 10.0.0.161..
2026-08-03 05:09:14,922 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9?flat
2026-08-03 05:09:14,943 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-6129-1']
2026-08-03 05:09:14,944 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-1:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:09:15,008 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-1?flat
2026-08-03 05:09:15,068 - DEBUG - VM 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-1 uve
2026-08-03 05:09:15,068 - DEBUG - Validated that VM ctest-vn1_vm2-26521843 is in Vrouter an-jenkins-deploy-platform-ansible-os-6129-1 UVE
2026-08-03 05:09:15,068 - DEBUG - Verifying vm tap interface/vn in vrouter uve
2026-08-03 05:09:15,079 - DEBUG - Expected tap interface of VM uuid 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 is default-domain:ctest-TestBasicVMVN0-48746621:06e93982-f5ea-4ddf-82b8-dd1ed9b4fb4a
2026-08-03 05:09:15,079 - DEBUG - Expected VN of VM uuid 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 is default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142
2026-08-03 05:09:15,079 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-1:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:09:15,166 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-1?flat
2026-08-03 05:09:15,212 - DEBUG - VM 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-1 uve
2026-08-03 05:09:15,213 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-48746621:06e93982-f5ea-4ddf-82b8-dd1ed9b4fb4a of vm 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-1 uve
2026-08-03 05:09:15,213 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 of vm 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-1 uve
2026-08-03 05:09:15,213 - DEBUG - Verifying the 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 virtual network link through opserver 10.0.0.249
2026-08-03 05:09:15,213 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machines
2026-08-03 05:09:15,230 - DEBUG - VM link and name as {'name': '7541dfc5-c4be-4082-b6ad-ab2e70520fd9', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9?flat'}
2026-08-03 05:09:15,230 - DEBUG - Verifying the 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 virtual network link through opserver 10.0.0.50
2026-08-03 05:09:15,230 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines
2026-08-03 05:09:15,291 - DEBUG - VM link and name as {'name': '7541dfc5-c4be-4082-b6ad-ab2e70520fd9', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9?flat'}
2026-08-03 05:09:15,291 - DEBUG - Verifying the 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 virtual network link through opserver 10.0.0.161
2026-08-03 05:09:15,291 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machines
2026-08-03 05:09:15,306 - DEBUG - VM link and name as {'name': '7541dfc5-c4be-4082-b6ad-ab2e70520fd9', 'href': 'http://10.0.0.161:8081/analytics/uves/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9?flat'}
2026-08-03 05:09:15,306 - INFO - VM ctest-vn1_vm2-26521843 validations in Opserver passed
2026-08-03 05:09:15,307 - INFO - Checking the ping between the VM again
2026-08-03 05:09:15,307 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:09:15,307 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123
2026-08-03 05:09:15,307 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4
2026-08-03 05:09:17,616 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=6.33 ms
64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.78 ms
64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.38 ms
--- 192.168.1.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2005ms
rtt min/avg/max/mdev = 1.381/3.165/6.332/2.245 ms
2026-08-03 05:09:17,616 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-80444148 passed
2026-08-03 05:09:17,616 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:09:17,616 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.38, gateway password: c0ntrail123
2026-08-03 05:09:17,616 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3
2026-08-03 05:09:19,879 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=2.98 ms
64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.81 ms
64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.74 ms
--- 192.168.1.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.741/2.178/2.984/0.571 ms
2026-08-03 05:09:19,879 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-26521843 passed
2026-08-03 05:09:19,879 - INFO - Deleting VM ctest-vn1_vm2-26521843
2026-08-03 05:09:19,970 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:09:19,970 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9
2026-08-03 05:09:19,983 - DEBUG - VM ID 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 of VM ctest-vn1_vm2-26521843 is still found in API Server
2026-08-03 05:09:21,984 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:09:21,984 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9
2026-08-03 05:09:21,999 - DEBUG - VM ID 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 of VM ctest-vn1_vm2-26521843 is still found in API Server
2026-08-03 05:09:23,999 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:09:24,000 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9
2026-08-03 05:09:24,009 - DEBUG - Response Code: 404
2026-08-03 05:09:24,010 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9
2026-08-03 05:09:24,019 - DEBUG - Response Code: 404
2026-08-03 05:09:24,019 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/7541dfc5-c4be-4082-b6ad-ab2e70520fd9
2026-08-03 05:09:24,028 - DEBUG - Response Code: 404
2026-08-03 05:09:24,029 - INFO - VM ctest-vn1_vm2-26521843 is fully removed in API-Server
2026-08-03 05:09:24,193 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found
2026-08-03 05:09:24,253 - DEBUG - Validated that vrouter 10.0.0.38 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142
2026-08-03 05:09:24,410 - DEBUG - VM route 192.168.1.4 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 of compute 10.0.0.19
2026-08-03 05:09:24,416 - DEBUG - Validated that vrouter 10.0.0.19 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142
2026-08-03 05:09:24,417 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142
2026-08-03 05:09:24,417 - INFO - VM ctest-vn1_vm2-26521843 is removed in Compute, and routes are removed in all compute nodes
2026-08-03 05:09:24,712 - INFO - Routes for VM ctest-vn1_vm2-26521843 is removed in all control-nodes
2026-08-03 05:09:24,712 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-1:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:09:24,768 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-1?flat
2026-08-03 05:09:24,809 - DEBUG - VM 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 is still present in vrouter an-jenkins-deploy-platform-ansible-os-6129-1 uve
2026-08-03 05:09:24,809 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2026-08-03 05:09:24,809 - DEBUG - interface for vm 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 deleted from vrouter uve ...
2026-08-03 05:09:24,809 - DEBUG - Deleted interface not in error interface list ...
2026-08-03 05:09:27,810 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-1:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:09:27,873 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-1?flat
2026-08-03 05:09:27,914 - DEBUG - VM 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 is still present in vrouter an-jenkins-deploy-platform-ansible-os-6129-1 uve
2026-08-03 05:09:27,914 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2026-08-03 05:09:27,914 - DEBUG - interface for vm 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 deleted from vrouter uve ...
2026-08-03 05:09:27,914 - DEBUG - Deleted interface not in error interface list ...
2026-08-03 05:09:30,915 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-1:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:09:30,983 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-1?flat
2026-08-03 05:09:31,023 - DEBUG - VM 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 is still present in vrouter an-jenkins-deploy-platform-ansible-os-6129-1 uve
2026-08-03 05:09:31,023 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2026-08-03 05:09:31,024 - DEBUG - interface for vm 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 deleted from vrouter uve ...
2026-08-03 05:09:31,024 - DEBUG - Deleted interface not in error interface list ...
2026-08-03 05:09:34,024 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-1:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:09:34,090 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-1?flat
2026-08-03 05:09:34,129 - DEBUG - VM 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 is not present in vrouter an-jenkins-deploy-platform-ansible-os-6129-1 uve
2026-08-03 05:09:34,129 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2026-08-03 05:09:34,129 - DEBUG - interface for vm 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 deleted from vrouter uve ...
2026-08-03 05:09:34,129 - DEBUG - Deleted interface not in error interface list ...
2026-08-03 05:09:34,129 - DEBUG - Validated that VM 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 is removed in Vrouter UVE
2026-08-03 05:09:34,129 - DEBUG - Verifying through opserver in 10.0.0.249
2026-08-03 05:09:34,129 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142?flat
2026-08-03 05:09:34,163 - DEBUG - 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 BM not in default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 uve
2026-08-03 05:09:34,163 - DEBUG - Validated that VM 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 is not present in the VN UVE
2026-08-03 05:09:34,163 - DEBUG - Verifying the 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 virtual network link through opserver 10.0.0.249
2026-08-03 05:09:34,163 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machines
2026-08-03 05:09:34,179 - INFO - 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 vm uve deleted from opserver
2026-08-03 05:09:34,180 - DEBUG - Verifying the 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 virtual network link through opserver 10.0.0.50
2026-08-03 05:09:34,180 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines
2026-08-03 05:09:34,195 - INFO - 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 vm uve deleted from opserver
2026-08-03 05:09:34,195 - DEBUG - Verifying the 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 virtual network link through opserver 10.0.0.161
2026-08-03 05:09:34,196 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machines
2026-08-03 05:09:34,212 - INFO - 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 vm uve deleted from opserver
2026-08-03 05:09:34,212 - INFO - Validated that VM 7541dfc5-c4be-4082-b6ad-ab2e70520fd9 is removed from Opserver
2026-08-03 05:09:34,212 - INFO - Deleting VM ctest-vn1_vm1-80444148
2026-08-03 05:09:34,301 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:09:34,301 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce
2026-08-03 05:09:34,317 - DEBUG - VM ID f7ecf15c-6909-44e8-98a2-19d1ead5fbce of VM ctest-vn1_vm1-80444148 is still found in API Server
2026-08-03 05:09:36,318 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:09:36,318 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce
2026-08-03 05:09:36,330 - DEBUG - VM ID f7ecf15c-6909-44e8-98a2-19d1ead5fbce of VM ctest-vn1_vm1-80444148 is still found in API Server
2026-08-03 05:09:38,331 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:09:38,331 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce
2026-08-03 05:09:38,341 - DEBUG - Response Code: 404
2026-08-03 05:09:38,341 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce
2026-08-03 05:09:38,350 - DEBUG - Response Code: 404
2026-08-03 05:09:38,350 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/f7ecf15c-6909-44e8-98a2-19d1ead5fbce
2026-08-03 05:09:38,358 - DEBUG - Response Code: 404
2026-08-03 05:09:38,358 - INFO - VM ctest-vn1_vm1-80444148 is fully removed in API-Server
2026-08-03 05:09:38,521 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found
2026-08-03 05:09:38,588 - DEBUG - Validated that vrouter 10.0.0.38 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142
2026-08-03 05:09:38,607 - DEBUG - Validated that vrouter 10.0.0.19 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142
2026-08-03 05:09:38,608 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142
2026-08-03 05:09:38,608 - INFO - VM ctest-vn1_vm1-80444148 is removed in Compute, and routes are removed in all compute nodes
2026-08-03 05:09:38,862 - INFO - Routes for VM ctest-vn1_vm1-80444148 is removed in all control-nodes
2026-08-03 05:09:38,863 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-2:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:09:38,911 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-2?flat
2026-08-03 05:09:38,948 - DEBUG - VM f7ecf15c-6909-44e8-98a2-19d1ead5fbce is still present in vrouter an-jenkins-deploy-platform-ansible-os-6129-2 uve
2026-08-03 05:09:38,948 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2026-08-03 05:09:38,948 - DEBUG - interface for vm f7ecf15c-6909-44e8-98a2-19d1ead5fbce deleted from vrouter uve ...
2026-08-03 05:09:38,948 - DEBUG - Deleted interface not in error interface list ...
2026-08-03 05:09:41,949 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-2:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:09:41,994 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-2?flat
2026-08-03 05:09:42,035 - DEBUG - VM f7ecf15c-6909-44e8-98a2-19d1ead5fbce is not present in vrouter an-jenkins-deploy-platform-ansible-os-6129-2 uve
2026-08-03 05:09:42,035 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2026-08-03 05:09:42,035 - DEBUG - interface for vm f7ecf15c-6909-44e8-98a2-19d1ead5fbce deleted from vrouter uve ...
2026-08-03 05:09:42,035 - DEBUG - Deleted interface not in error interface list ...
2026-08-03 05:09:42,035 - DEBUG - Validated that VM f7ecf15c-6909-44e8-98a2-19d1ead5fbce is removed in Vrouter UVE
2026-08-03 05:09:42,035 - DEBUG - Verifying through opserver in 10.0.0.249
2026-08-03 05:09:42,035 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142?flat
2026-08-03 05:09:42,059 - DEBUG - f7ecf15c-6909-44e8-98a2-19d1ead5fbce BM not in default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-49411142 uve
2026-08-03 05:09:42,059 - DEBUG - Validated that VM f7ecf15c-6909-44e8-98a2-19d1ead5fbce is not present in the VN UVE
2026-08-03 05:09:42,059 - DEBUG - Verifying the f7ecf15c-6909-44e8-98a2-19d1ead5fbce virtual network link through opserver 10.0.0.249
2026-08-03 05:09:42,059 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machines
2026-08-03 05:09:42,075 - INFO - f7ecf15c-6909-44e8-98a2-19d1ead5fbce vm uve deleted from opserver
2026-08-03 05:09:42,075 - DEBUG - Verifying the f7ecf15c-6909-44e8-98a2-19d1ead5fbce virtual network link through opserver 10.0.0.50
2026-08-03 05:09:42,075 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines
2026-08-03 05:09:42,091 - INFO - f7ecf15c-6909-44e8-98a2-19d1ead5fbce vm uve deleted from opserver
2026-08-03 05:09:42,091 - DEBUG - Verifying the f7ecf15c-6909-44e8-98a2-19d1ead5fbce virtual network link through opserver 10.0.0.161
2026-08-03 05:09:42,091 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machines
2026-08-03 05:09:42,109 - INFO - f7ecf15c-6909-44e8-98a2-19d1ead5fbce vm uve deleted from opserver
2026-08-03 05:09:42,109 - INFO - Validated that VM f7ecf15c-6909-44e8-98a2-19d1ead5fbce is removed from Opserver
2026-08-03 05:09:42,109 - INFO - Deleting VN ctest-vn1-49411142
2026-08-03 05:09:42,366 - DEBUG - Response for deleting network ()
2026-08-03 05:09:42,367 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/63451959-1b72-4828-9be7-8a838a487dab
2026-08-03 05:09:42,374 - DEBUG - Response Code: 404
2026-08-03 05:09:42,374 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2026-08-03 05:09:42,381 - DEBUG - Requesting: http://10.0.0.249:8082/domain/724de60e-9f87-4d23-867c-6bd9a9880f5b
2026-08-03 05:09:42,441 - DEBUG - Requesting: http://10.0.0.249:8082/project/34ffcb1a-09fd-4528-8d5b-a11d79da58f5
2026-08-03 05:09:42,546 - INFO - Validated that VN ctest-vn1-49411142 is not found in API Server
2026-08-03 05:09:42,560 - DEBUG - VN ctest-vn1-49411142 is not present in Agent 10.0.0.38
2026-08-03 05:09:42,575 - DEBUG - VN ctest-vn1-49411142 is not present in Agent 10.0.0.19
2026-08-03 05:09:42,575 - INFO - Validated that VN ctest-vn1-49411142 is not in any agent
2026-08-03 05:09:42,617 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-49411142 info
2026-08-03 05:09:43,946 - DEBUG - Skipping xmpp flap check
2026-08-03 05:09:43,946 - INFO - END TEST : test_control_node_switchover : PASSED[0:04:19]
2026-08-03 05:09:43,946 - INFO - --------------------------------------------------------------------------------
2026-08-03 05:09:43,950 - INFO - ================================================================================
2026-08-03 05:09:43,950 - INFO - STARTING TEST : test_process_restart_in_policy_between_vns
2026-08-03 05:09:43,950 - INFO - TEST DESCRIPTION : Test to validate that with policy having rule to check icmp fwding between VMs on different VNs , ping between VMs should pass
with process restarts
1. Pick 2 VN's from resource pool which has one VM each
2. Create policy with icmp allow rule between those VN's and bind it networks
3. Ping from one VM to another VM
4. Restart process 'vrouter' and 'control' on setup
5. Ping again between VM's after process restart
Pass criteria: Step 2,3,4 and 5 should pass
2026-08-03 05:09:45,273 - DEBUG - Skipping xmpp flap check
2026-08-03 05:09:45,274 - INFO - Initial checks done. Running the testcase now
2026-08-03 05:09:45,274 - INFO -
2026-08-03 05:09:45,282 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None]
2026-08-03 05:09:45,647 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None]
2026-08-03 05:09:46,391 - INFO - Created VN ctest-vn1-43687081, UUID :037fe6af-1c0e-4e8c-98f0-f57c25b33b36
2026-08-03 05:09:46,801 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2026-08-03 05:09:46,810 - DEBUG - Requesting: http://10.0.0.249:8082/domain/724de60e-9f87-4d23-867c-6bd9a9880f5b
2026-08-03 05:09:46,828 - DEBUG - Requesting: http://10.0.0.249:8082/project/34ffcb1a-09fd-4528-8d5b-a11d79da58f5
2026-08-03 05:09:46,947 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/037fe6af-1c0e-4e8c-98f0-f57c25b33b36
2026-08-03 05:09:46,964 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/037fe6af-1c0e-4e8c-98f0-f57c25b33b36
2026-08-03 05:09:46,976 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/dbe4fe15-976a-4496-9314-6c8fd9559164
2026-08-03 05:09:46,990 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/dbe4fe15-976a-4496-9314-6c8fd9559164
2026-08-03 05:09:46,999 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/c48e4bf4-4bf1-4397-8398-f7cb4cf592eb
2026-08-03 05:09:47,008 - DEBUG - Route Targets: ['target:64512:8000004']
2026-08-03 05:09:47,008 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/037fe6af-1c0e-4e8c-98f0-f57c25b33b36
2026-08-03 05:09:47,020 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/dbe4fe15-976a-4496-9314-6c8fd9559164
2026-08-03 05:09:47,043 - INFO - Verified VN network id 13 for VN 037fe6af-1c0e-4e8c-98f0-f57c25b33b36
2026-08-03 05:09:47,043 - INFO - Verifications in API Server for VN ctest-vn1-43687081 passed
2026-08-03 05:09:47,043 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/037fe6af-1c0e-4e8c-98f0-f57c25b33b36
2026-08-03 05:09:47,065 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/dbe4fe15-976a-4496-9314-6c8fd9559164
2026-08-03 05:09:47,076 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/dbe4fe15-976a-4496-9314-6c8fd9559164
2026-08-03 05:09:47,094 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/c48e4bf4-4bf1-4397-8398-f7cb4cf592eb
2026-08-03 05:09:47,113 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-48746621', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '252173743920402060', 'uuid-lslong': '11020578201732856630'}, 'enable': 'true', 'created': '2026-08-03T05:09:46', 'last-modified': '2026-08-03T05:09:46', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.539978'}
2026-08-03 05:09:47,124 - DEBUG - Route Targets: ['target:64512:8000004']
2026-08-03 05:09:47,131 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-48746621', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '252173743920402060', 'uuid-lslong': '11020578201732856630'}, 'enable': 'true', 'created': '2026-08-03T05:09:46', 'last-modified': '2026-08-03T05:09:46', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.567836'}
2026-08-03 05:09:47,138 - DEBUG - Route Targets: ['target:64512:8000004']
2026-08-03 05:09:47,146 - DEBUG - Control-node 10.0.0.161 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-48746621', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '252173743920402060', 'uuid-lslong': '11020578201732856630'}, 'enable': 'true', 'created': '2026-08-03T05:09:46', 'last-modified': '2026-08-03T05:09:46', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.573591'}
2026-08-03 05:09:47,154 - DEBUG - Route Targets: ['target:64512:8000004']
2026-08-03 05:09:47,155 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-43687081 passed
2026-08-03 05:09:47,155 - DEBUG - ====Verifying policy data for ctest-vn1-43687081 in API_Server ======
2026-08-03 05:09:47,155 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2026-08-03 05:09:47,165 - DEBUG - Requesting: http://10.0.0.249:8082/domain/724de60e-9f87-4d23-867c-6bd9a9880f5b
2026-08-03 05:09:47,179 - DEBUG - Requesting: http://10.0.0.249:8082/project/34ffcb1a-09fd-4528-8d5b-a11d79da58f5
2026-08-03 05:09:47,311 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/037fe6af-1c0e-4e8c-98f0-f57c25b33b36
2026-08-03 05:09:47,324 - DEBUG - =>VN ctest-vn1-43687081 has no policy to be verified
2026-08-03 05:09:47,324 - DEBUG - Verifying the vn in opserver
2026-08-03 05:09:47,324 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 virtual network link through opserver 10.0.0.249
2026-08-03 05:09:47,325 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks
2026-08-03 05:09:47,342 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081?flat'}
2026-08-03 05:09:47,342 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 virtual network link through opserver 10.0.0.50
2026-08-03 05:09:47,342 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks
2026-08-03 05:09:47,358 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081?flat'}
2026-08-03 05:09:47,358 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 virtual network link through opserver 10.0.0.161
2026-08-03 05:09:47,358 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-networks
2026-08-03 05:09:47,374 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'href': 'http://10.0.0.161:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081?flat'}
2026-08-03 05:09:47,374 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is found in opserver
2026-08-03 05:09:47,415 - DEBUG - Do not have enough data to verify VN in agent
2026-08-03 05:09:47,427 - DEBUG - VRF ids for VN ctest-vn1-43687081: {}
2026-08-03 05:09:48,275 - INFO - Created VN ctest-vn2-81446240, UUID :a7c4faf9-ef5f-4c0c-b87e-bbeec2a5ab08
2026-08-03 05:09:48,710 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2026-08-03 05:09:48,718 - DEBUG - Requesting: http://10.0.0.249:8082/domain/724de60e-9f87-4d23-867c-6bd9a9880f5b
2026-08-03 05:09:48,732 - DEBUG - Requesting: http://10.0.0.249:8082/project/34ffcb1a-09fd-4528-8d5b-a11d79da58f5
2026-08-03 05:09:48,849 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/a7c4faf9-ef5f-4c0c-b87e-bbeec2a5ab08
2026-08-03 05:09:48,868 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/a7c4faf9-ef5f-4c0c-b87e-bbeec2a5ab08
2026-08-03 05:09:48,882 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/7b24b36a-bcc8-4688-a72a-458ce9557d43
2026-08-03 05:09:48,894 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/7b24b36a-bcc8-4688-a72a-458ce9557d43
2026-08-03 05:09:48,904 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/f19528f7-3888-4492-84d3-726b5d765347
2026-08-03 05:09:48,915 - DEBUG - Route Targets: ['target:64512:8000006']
2026-08-03 05:09:48,915 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/a7c4faf9-ef5f-4c0c-b87e-bbeec2a5ab08
2026-08-03 05:09:48,928 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/7b24b36a-bcc8-4688-a72a-458ce9557d43
2026-08-03 05:09:48,955 - INFO - Verified VN network id 14 for VN a7c4faf9-ef5f-4c0c-b87e-bbeec2a5ab08
2026-08-03 05:09:48,955 - INFO - Verifications in API Server for VN ctest-vn2-81446240 passed
2026-08-03 05:09:48,955 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/a7c4faf9-ef5f-4c0c-b87e-bbeec2a5ab08
2026-08-03 05:09:48,969 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/7b24b36a-bcc8-4688-a72a-458ce9557d43
2026-08-03 05:09:48,978 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/7b24b36a-bcc8-4688-a72a-458ce9557d43
2026-08-03 05:09:48,988 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/f19528f7-3888-4492-84d3-726b5d765347
2026-08-03 05:09:49,006 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-48746621', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12089063251139054604', 'uuid-lslong': '13294269784186530568'}, 'enable': 'true', 'created': '2026-08-03T05:09:48', 'last-modified': '2026-08-03T05:09:48', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.535138'}
2026-08-03 05:09:49,014 - DEBUG - Route Targets: ['target:64512:8000006']
2026-08-03 05:09:49,020 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-48746621', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12089063251139054604', 'uuid-lslong': '13294269784186530568'}, 'enable': 'true', 'created': '2026-08-03T05:09:48', 'last-modified': '2026-08-03T05:09:48', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.550839'}
2026-08-03 05:09:49,028 - DEBUG - Route Targets: ['target:64512:8000006']
2026-08-03 05:09:49,036 - DEBUG - Control-node 10.0.0.161 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-48746621', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12089063251139054604', 'uuid-lslong': '13294269784186530568'}, 'enable': 'true', 'created': '2026-08-03T05:09:48', 'last-modified': '2026-08-03T05:09:48', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.565448'}
2026-08-03 05:09:49,045 - DEBUG - Route Targets: ['target:64512:8000006']
2026-08-03 05:09:49,045 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-81446240 passed
2026-08-03 05:09:49,045 - DEBUG - ====Verifying policy data for ctest-vn2-81446240 in API_Server ======
2026-08-03 05:09:49,045 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2026-08-03 05:09:49,053 - DEBUG - Requesting: http://10.0.0.249:8082/domain/724de60e-9f87-4d23-867c-6bd9a9880f5b
2026-08-03 05:09:49,071 - DEBUG - Requesting: http://10.0.0.249:8082/project/34ffcb1a-09fd-4528-8d5b-a11d79da58f5
2026-08-03 05:09:49,183 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/a7c4faf9-ef5f-4c0c-b87e-bbeec2a5ab08
2026-08-03 05:09:49,199 - DEBUG - =>VN ctest-vn2-81446240 has no policy to be verified
2026-08-03 05:09:49,199 - DEBUG - Verifying the vn in opserver
2026-08-03 05:09:49,199 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 virtual network link through opserver 10.0.0.249
2026-08-03 05:09:49,199 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks
2026-08-03 05:09:49,217 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240?flat'}
2026-08-03 05:09:49,218 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 virtual network link through opserver 10.0.0.50
2026-08-03 05:09:49,218 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks
2026-08-03 05:09:49,247 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240?flat'}
2026-08-03 05:09:49,247 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 virtual network link through opserver 10.0.0.161
2026-08-03 05:09:49,248 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-networks
2026-08-03 05:09:49,266 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240', 'href': 'http://10.0.0.161:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240?flat'}
2026-08-03 05:09:49,266 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 is found in opserver
2026-08-03 05:09:49,294 - DEBUG - Do not have enough data to verify VN in agent
2026-08-03 05:09:49,313 - DEBUG - VRF ids for VN ctest-vn2-81446240: {}
2026-08-03 05:09:50,592 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-6129-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-6129-2)
2026-08-03 05:09:51,350 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-6129-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-6129-1)
2026-08-03 05:09:51,350 - INFO - Waiting for VM ctest-vn1_vm1-46511698 to be up..
2026-08-03 05:09:51,434 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:09:56,563 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:10:01,683 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:10:06,786 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:10:11,908 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:10:17,048 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:10:22,151 - DEBUG - VM is in ACTIVE state now
2026-08-03 05:10:22,151 - INFO - VM name : ctest-vn1_vm1-46511698
2026-08-03 05:10:22,274 - DEBUG - VM ctest-vn1_vm1-46511698 ID is 2ee71b48-3fe8-484a-a055-b002968f1362
2026-08-03 05:10:22,274 - DEBUG - VM ctest-vn1_vm1-46511698 launched on Node an-jenkins-deploy-platform-ansible-os-6129-2
2026-08-03 05:10:22,379 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/2ee71b48-3fe8-484a-a055-b002968f1362
2026-08-03 05:10:22,394 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/04227adc-3efd-4c4c-9d97-17325e13a5bd
2026-08-03 05:10:25,768 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1028ms')
2026-08-03 05:10:25,769 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-46511698 failed!
2026-08-03 05:10:25,815 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:10:29,899 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms')
2026-08-03 05:10:29,899 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-46511698 failed!
2026-08-03 05:10:29,919 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:10:33,986 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1004ms')
2026-08-03 05:10:33,987 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-46511698 failed!
2026-08-03 05:10:34,009 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:10:38,088 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms')
2026-08-03 05:10:38,088 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-46511698 failed!
2026-08-03 05:10:38,111 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:10:42,182 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1005ms')
2026-08-03 05:10:42,182 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-46511698 failed!
2026-08-03 05:10:42,204 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:10:46,277 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1008ms')
2026-08-03 05:10:46,277 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-46511698 failed!
2026-08-03 05:10:46,298 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:10:50,374 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms')
2026-08-03 05:10:50,374 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-46511698 failed!
2026-08-03 05:10:50,401 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:10:54,502 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms')
2026-08-03 05:10:54,502 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-46511698 failed!
2026-08-03 05:10:54,524 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:10:58,600 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=9.82 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1005ms\r\nrtt min/avg/max/mdev = 9.823/9.823/9.823/0.000 ms')
2026-08-03 05:10:58,600 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-46511698 passed
2026-08-03 05:10:58,691 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:10:58,691 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-46511698, IP 192.168.1.252, Port 22
2026-08-03 05:10:58,783 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2026-08-03 05:10:58,958 - DEBUG - VM ctest-vn1_vm1-46511698 is NOT ready for SSH connections, VM status: ACTIVE
2026-08-03 05:11:03,959 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:11:03,960 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-46511698, IP 192.168.1.252, Port 22
2026-08-03 05:11:04,032 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2026-08-03 05:11:04,159 - DEBUG - VM ctest-vn1_vm1-46511698 is NOT ready for SSH connections, VM status: ACTIVE
2026-08-03 05:11:09,159 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:11:09,160 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-46511698, IP 192.168.1.252, Port 22
2026-08-03 05:11:09,226 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2026-08-03 05:11:09,355 - DEBUG - VM ctest-vn1_vm1-46511698 is NOT ready for SSH connections, VM status: ACTIVE
2026-08-03 05:11:14,356 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:11:14,356 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-46511698, IP 192.168.1.252, Port 22
2026-08-03 05:11:14,530 - DEBUG - VM ctest-vn1_vm1-46511698 is ready for SSH connections
2026-08-03 05:11:14,530 - INFO - Waiting for VM ctest-vn2_vm1-81867373 to be up..
2026-08-03 05:11:14,633 - DEBUG - VM is in ACTIVE state now
2026-08-03 05:11:14,633 - INFO - VM name : ctest-vn2_vm1-81867373
2026-08-03 05:11:14,742 - DEBUG - VM ctest-vn2_vm1-81867373 ID is be477428-4431-411c-a0da-5abdec8026a0
2026-08-03 05:11:14,742 - DEBUG - VM ctest-vn2_vm1-81867373 launched on Node an-jenkins-deploy-platform-ansible-os-6129-1
2026-08-03 05:11:14,851 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/be477428-4431-411c-a0da-5abdec8026a0
2026-08-03 05:11:14,895 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/3696ad97-90fa-4d83-8d2b-361dd76c08ef
2026-08-03 05:11:16,102 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=4.06 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.704 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.704/2.380/4.056/1.676 ms')
2026-08-03 05:11:16,102 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm1-81867373 passed
2026-08-03 05:11:16,188 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:11:16,188 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-81867373, IP 192.168.2.252, Port 22
2026-08-03 05:11:16,364 - DEBUG - VM ctest-vn2_vm1-81867373 is ready for SSH connections
2026-08-03 05:11:16,364 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:11:16,364 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.252, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123
2026-08-03 05:11:16,364 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.252
2026-08-03 05:11:28,059 - DEBUG - PING 192.168.2.252 (192.168.2.252) 56(84) bytes of data.
64 bytes from 192.168.2.252: icmp_req=1 ttl=63 time=11.0 ms
64 bytes from 192.168.2.252: icmp_req=2 ttl=63 time=4.62 ms
64 bytes from 192.168.2.252: icmp_req=3 ttl=63 time=5.04 ms
--- 192.168.2.252 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 4.625/6.906/11.046/2.932 ms
2026-08-03 05:11:28,060 - INFO - Ping to IP 192.168.2.252 from VM ctest-vn1_vm1-46511698 passed
2026-08-03 05:11:31,523 - INFO - 10.0.0.38
2026-08-03 05:11:31,528 - INFO - agent:initializing
2026-08-03 05:11:31,528 - DEBUG - defaultdict(, {'10.0.0.38': {'agent': {'status': 'initializing', 'description': None}}})
2026-08-03 05:11:31,528 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2026-08-03 05:11:36,632 - INFO - 10.0.0.38
2026-08-03 05:11:36,640 - INFO - agent:active
2026-08-03 05:11:36,640 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.38
2026-08-03 05:11:40,211 - INFO - 10.0.0.19
2026-08-03 05:11:40,217 - INFO - agent:initializing
2026-08-03 05:11:40,217 - DEBUG - defaultdict(, {'10.0.0.19': {'agent': {'status': 'initializing', 'description': None}}})
2026-08-03 05:11:40,217 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2026-08-03 05:11:45,317 - INFO - 10.0.0.19
2026-08-03 05:11:45,326 - INFO - agent:active
2026-08-03 05:11:45,326 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.19
2026-08-03 05:11:46,151 - INFO - 10.0.0.249
2026-08-03 05:11:46,156 - INFO - control:initializing
2026-08-03 05:11:46,156 - DEBUG - defaultdict(, {'10.0.0.249': {'control': {'status': 'initializing', 'description': None}}})
2026-08-03 05:11:46,156 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2026-08-03 05:11:51,281 - INFO - 10.0.0.249
2026-08-03 05:11:51,296 - INFO - control:active
2026-08-03 05:11:51,296 - INFO - Contrail services ['control'] are up on nodes 10.0.0.249
2026-08-03 05:11:51,719 - INFO - 10.0.0.50
2026-08-03 05:11:51,724 - INFO - control:initializing
2026-08-03 05:11:51,724 - DEBUG - defaultdict(, {'10.0.0.50': {'control': {'status': 'initializing', 'description': None}}})
2026-08-03 05:11:51,724 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2026-08-03 05:11:56,826 - INFO - 10.0.0.50
2026-08-03 05:11:56,834 - INFO - control:active
2026-08-03 05:11:56,835 - INFO - Contrail services ['control'] are up on nodes 10.0.0.50
2026-08-03 05:11:57,385 - INFO - 10.0.0.161
2026-08-03 05:11:57,389 - INFO - control:initializing
2026-08-03 05:11:57,389 - DEBUG - defaultdict(, {'10.0.0.161': {'control': {'status': 'initializing', 'description': None}}})
2026-08-03 05:11:57,389 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2026-08-03 05:12:02,550 - INFO - 10.0.0.161
2026-08-03 05:12:02,559 - INFO - control:active
2026-08-03 05:12:02,559 - INFO - Contrail services ['control'] are up on nodes 10.0.0.161
2026-08-03 05:12:03,541 - INFO - 10.0.0.249
2026-08-03 05:12:03,546 - INFO - api-server:initializing
2026-08-03 05:12:03,546 - DEBUG - defaultdict(, {'10.0.0.249': {'api-server': {'status': 'initializing', 'description': None}}})
2026-08-03 05:12:03,546 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2026-08-03 05:12:08,666 - INFO - 10.0.0.249
2026-08-03 05:12:08,670 - INFO - api-server:initializing
2026-08-03 05:12:08,671 - DEBUG - defaultdict(, {'10.0.0.249': {'api-server': {'status': 'initializing', 'description': None}}})
2026-08-03 05:12:08,671 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1
2026-08-03 05:12:13,834 - INFO - 10.0.0.249
2026-08-03 05:12:13,840 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down)
2026-08-03 05:12:13,841 - DEBUG - defaultdict(, {'10.0.0.249': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}})
2026-08-03 05:12:13,841 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2
2026-08-03 05:12:19,035 - INFO - 10.0.0.249
2026-08-03 05:12:19,041 - INFO - api-server:active
2026-08-03 05:12:19,041 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.249
2026-08-03 05:12:19,794 - INFO - 10.0.0.50
2026-08-03 05:12:19,798 - INFO - api-server:initializing
2026-08-03 05:12:19,798 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': None}}})
2026-08-03 05:12:19,799 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2026-08-03 05:12:24,903 - INFO - 10.0.0.50
2026-08-03 05:12:24,907 - INFO - api-server:initializing
2026-08-03 05:12:24,907 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': None}}})
2026-08-03 05:12:24,908 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1
2026-08-03 05:12:30,007 - INFO - 10.0.0.50
2026-08-03 05:12:30,013 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down)
2026-08-03 05:12:30,013 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}})
2026-08-03 05:12:30,013 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2
2026-08-03 05:12:35,119 - INFO - 10.0.0.50
2026-08-03 05:12:35,125 - INFO - api-server:active
2026-08-03 05:12:35,126 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.50
2026-08-03 05:12:36,135 - INFO - 10.0.0.161
2026-08-03 05:12:36,141 - INFO - api-server:initializing
2026-08-03 05:12:36,141 - DEBUG - defaultdict(, {'10.0.0.161': {'api-server': {'status': 'initializing', 'description': None}}})
2026-08-03 05:12:36,141 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2026-08-03 05:12:41,285 - INFO - 10.0.0.161
2026-08-03 05:12:41,290 - INFO - api-server:initializing
2026-08-03 05:12:41,290 - DEBUG - defaultdict(, {'10.0.0.161': {'api-server': {'status': 'initializing', 'description': None}}})
2026-08-03 05:12:41,291 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1
2026-08-03 05:12:46,413 - INFO - 10.0.0.161
2026-08-03 05:12:46,423 - INFO - api-server:initializing
2026-08-03 05:12:46,424 - DEBUG - defaultdict(, {'10.0.0.161': {'api-server': {'status': 'initializing', 'description': None}}})
2026-08-03 05:12:46,424 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2
2026-08-03 05:12:51,597 - INFO - 10.0.0.161
2026-08-03 05:12:51,617 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart] connection down)
2026-08-03 05:12:51,617 - DEBUG - defaultdict(, {'10.0.0.161': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart] connection down'}}})
2026-08-03 05:12:51,617 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 3
2026-08-03 05:12:56,751 - INFO - 10.0.0.161
2026-08-03 05:12:56,758 - INFO - api-server:active
2026-08-03 05:12:56,758 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.161
2026-08-03 05:12:56,900 - INFO - 10.0.0.249
2026-08-03 05:12:56,900 - INFO - config
2026-08-03 05:12:56,906 - INFO - config-nodemgr:active
2026-08-03 05:12:56,911 - INFO - api-server:active
2026-08-03 05:12:56,916 - INFO - schema:backup
2026-08-03 05:12:56,921 - INFO - svc-monitor:backup
2026-08-03 05:12:56,926 - INFO - device-manager:backup
2026-08-03 05:12:56,927 - INFO - config-database
2026-08-03 05:12:56,927 - INFO - config-cassandra:active
2026-08-03 05:12:56,928 - INFO - config-zookeeper:active
2026-08-03 05:12:56,928 - INFO - config-rabbitmq:active
2026-08-03 05:12:56,929 - INFO - control
2026-08-03 05:12:56,938 - INFO - control-nodemgr:active
2026-08-03 05:12:56,949 - INFO - control:active
2026-08-03 05:12:56,949 - INFO - named:active
2026-08-03 05:12:56,960 - INFO - dns:active
2026-08-03 05:12:56,961 - INFO - analytics
2026-08-03 05:12:56,968 - INFO - analytics-nodemgr:active
2026-08-03 05:12:56,974 - INFO - analytics-api:active
2026-08-03 05:12:56,980 - INFO - collector:active
2026-08-03 05:12:56,980 - INFO - analytics-database
2026-08-03 05:12:56,980 - INFO - analytics-cassandra:active
2026-08-03 05:12:56,988 - INFO - analyticsdb-nodemgr:active
2026-08-03 05:12:56,997 - INFO - query-engine:active
2026-08-03 05:12:56,997 - INFO - webui
2026-08-03 05:12:56,997 - INFO - webui:active
2026-08-03 05:12:56,997 - INFO - webui-middleware:active
2026-08-03 05:12:56,998 - INFO - redis:active
2026-08-03 05:12:56,998 - INFO - analytics_snmp
2026-08-03 05:12:57,004 - INFO - snmp-collector:active
2026-08-03 05:12:57,005 - INFO - snmp-topology:active
2026-08-03 05:12:57,005 - INFO - analytics_alarm
2026-08-03 05:12:57,005 - INFO - alarmgen:active
2026-08-03 05:12:57,113 - INFO - 10.0.0.50
2026-08-03 05:12:57,114 - INFO - config
2026-08-03 05:12:57,120 - INFO - config-nodemgr:active
2026-08-03 05:12:57,126 - INFO - api-server:active
2026-08-03 05:12:57,131 - INFO - schema:active
2026-08-03 05:12:57,136 - INFO - svc-monitor:active
2026-08-03 05:12:57,142 - INFO - device-manager:active
2026-08-03 05:12:57,143 - INFO - config-database
2026-08-03 05:12:57,144 - INFO - config-cassandra:active
2026-08-03 05:12:57,144 - INFO - config-zookeeper:active
2026-08-03 05:12:57,145 - INFO - config-rabbitmq:active
2026-08-03 05:12:57,145 - INFO - control
2026-08-03 05:12:57,154 - INFO - control-nodemgr:active
2026-08-03 05:12:57,163 - INFO - control:active
2026-08-03 05:12:57,164 - INFO - named:active
2026-08-03 05:12:57,172 - INFO - dns:active
2026-08-03 05:12:57,172 - INFO - analytics
2026-08-03 05:12:57,180 - INFO - analytics-nodemgr:active
2026-08-03 05:12:57,185 - INFO - analytics-api:active
2026-08-03 05:12:57,192 - INFO - collector:active
2026-08-03 05:12:57,192 - INFO - analytics-database
2026-08-03 05:12:57,192 - INFO - analytics-cassandra:active
2026-08-03 05:12:57,198 - INFO - analyticsdb-nodemgr:active
2026-08-03 05:12:57,205 - INFO - query-engine:active
2026-08-03 05:12:57,205 - INFO - webui
2026-08-03 05:12:57,205 - INFO - webui:active
2026-08-03 05:12:57,205 - INFO - webui-middleware:active
2026-08-03 05:12:57,205 - INFO - redis:active
2026-08-03 05:12:57,205 - INFO - analytics_snmp
2026-08-03 05:12:57,211 - INFO - snmp-collector:active
2026-08-03 05:12:57,211 - INFO - snmp-topology:active
2026-08-03 05:12:57,212 - INFO - analytics_alarm
2026-08-03 05:12:57,212 - INFO - alarmgen:active
2026-08-03 05:12:57,332 - INFO - 10.0.0.161
2026-08-03 05:12:57,332 - INFO - config
2026-08-03 05:12:57,340 - INFO - config-nodemgr:active
2026-08-03 05:12:57,349 - INFO - api-server:active
2026-08-03 05:12:57,353 - INFO - schema:backup
2026-08-03 05:12:57,365 - INFO - svc-monitor:backup
2026-08-03 05:12:57,369 - INFO - device-manager:backup
2026-08-03 05:12:57,369 - INFO - config-database
2026-08-03 05:12:57,369 - INFO - config-cassandra:active
2026-08-03 05:12:57,369 - INFO - config-zookeeper:active
2026-08-03 05:12:57,369 - INFO - config-rabbitmq:active
2026-08-03 05:12:57,369 - INFO - control
2026-08-03 05:12:57,377 - INFO - control-nodemgr:active
2026-08-03 05:12:57,385 - INFO - control:active
2026-08-03 05:12:57,385 - INFO - named:active
2026-08-03 05:12:57,392 - INFO - dns:active
2026-08-03 05:12:57,392 - INFO - analytics
2026-08-03 05:12:57,401 - INFO - analytics-nodemgr:active
2026-08-03 05:12:57,409 - INFO - analytics-api:active
2026-08-03 05:12:57,418 - INFO - collector:active
2026-08-03 05:12:57,419 - INFO - analytics-database
2026-08-03 05:12:57,419 - INFO - analytics-cassandra:active
2026-08-03 05:12:57,427 - INFO - analyticsdb-nodemgr:active
2026-08-03 05:12:57,436 - INFO - query-engine:active
2026-08-03 05:12:57,436 - INFO - webui
2026-08-03 05:12:57,437 - INFO - webui:active
2026-08-03 05:12:57,437 - INFO - webui-middleware:active
2026-08-03 05:12:57,437 - INFO - redis:active
2026-08-03 05:12:57,437 - INFO - analytics_snmp
2026-08-03 05:12:57,444 - INFO - snmp-collector:active
2026-08-03 05:12:57,444 - INFO - snmp-topology:active
2026-08-03 05:12:57,444 - INFO - analytics_alarm
2026-08-03 05:12:57,444 - INFO - alarmgen:active
2026-08-03 05:12:57,544 - INFO - 10.0.0.38
2026-08-03 05:12:57,544 - INFO - vrouter
2026-08-03 05:12:57,551 - INFO - vrouter-nodemgr:active
2026-08-03 05:12:57,559 - INFO - agent:active
2026-08-03 05:12:57,662 - INFO - 10.0.0.19
2026-08-03 05:12:57,663 - INFO - vrouter
2026-08-03 05:12:57,671 - INFO - vrouter-nodemgr:active
2026-08-03 05:12:57,678 - INFO - agent:active
2026-08-03 05:12:57,679 - INFO - All the contrail services are up on all nodes
2026-08-03 05:12:57,679 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-1?flat
2026-08-03 05:12:58,172 - INFO - an-jenkins-deploy-platform-ansible-os-6129-1:contrail-vrouter-agent module connection to ['10.20.0.25:5269'] servers UP
2026-08-03 05:12:58,172 - INFO - an-jenkins-deploy-platform-ansible-os-6129-1 connected to xmpp 10.20.0.25
2026-08-03 05:12:58,172 - WARNING - an-jenkins-deploy-platform-ansible-os-6129-1:contrail-vrouter-agent module connection to ['10.20.0.29:5269'] servers NOT UP
2026-08-03 05:12:58,172 - INFO - an-jenkins-deploy-platform-ansible-os-6129-1:contrail-vrouter-agent module connection to ['10.20.0.30:5269'] servers UP
2026-08-03 05:12:58,172 - INFO - an-jenkins-deploy-platform-ansible-os-6129-1 connected to xmpp 10.20.0.30
2026-08-03 05:12:58,172 - INFO - an-jenkins-deploy-platform-ansible-os-6129-1:contrail-vrouter-agent module connection to ['10.20.0.25:53'] servers UP
2026-08-03 05:12:58,172 - INFO - an-jenkins-deploy-platform-ansible-os-6129-1 connected to dns 10.20.0.25
2026-08-03 05:12:58,172 - INFO - an-jenkins-deploy-platform-ansible-os-6129-1:contrail-vrouter-agent module connection to ['10.20.0.29:53'] servers UP
2026-08-03 05:12:58,172 - INFO - an-jenkins-deploy-platform-ansible-os-6129-1 connected to dns 10.20.0.29
2026-08-03 05:12:58,172 - WARNING - an-jenkins-deploy-platform-ansible-os-6129-1:contrail-vrouter-agent module connection to ['10.20.0.30:53'] servers NOT UP
2026-08-03 05:12:58,172 - INFO - an-jenkins-deploy-platform-ansible-os-6129-1:contrail-vrouter-agent module connection to ['10.0.0.249:8086'] servers UP
2026-08-03 05:12:58,172 - INFO - an-jenkins-deploy-platform-ansible-os-6129-1 connected to collector 10.0.0.249
2026-08-03 05:12:58,173 - WARNING - an-jenkins-deploy-platform-ansible-os-6129-1:contrail-vrouter-agent module connection to ['10.0.0.50:8086'] servers NOT UP
2026-08-03 05:12:58,173 - WARNING - an-jenkins-deploy-platform-ansible-os-6129-1:contrail-vrouter-agent module connection to ['10.0.0.161:8086'] servers NOT UP
2026-08-03 05:12:58,173 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-6129-2?flat
2026-08-03 05:12:58,644 - INFO - cn-jenkins-deploy-platform-ansible-os-6129-2:contrail-control module connection to 10.0.0.249:9041 servers UP
2026-08-03 05:12:58,644 - INFO - cn-jenkins-deploy-platform-ansible-os-6129-2:contrail-control module connection to 10.0.0.249:5673 servers UP
2026-08-03 05:12:58,645 - INFO - cn-jenkins-deploy-platform-ansible-os-6129-2:contrail-control module connection to ['10.0.0.249:8086'] servers UP
2026-08-03 05:12:58,645 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-6129-3?flat
2026-08-03 05:12:58,706 - INFO - cn-jenkins-deploy-platform-ansible-os-6129-3:contrail-control module connection to 10.0.0.249:9041 servers UP
2026-08-03 05:12:58,706 - INFO - cn-jenkins-deploy-platform-ansible-os-6129-3:contrail-control module connection to 10.0.0.249:5673 servers UP
2026-08-03 05:12:58,706 - INFO - cn-jenkins-deploy-platform-ansible-os-6129-3:contrail-control module connection to ['10.0.0.249:8086'] servers UP
2026-08-03 05:12:58,706 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-6129-30?flat
2026-08-03 05:12:58,728 - INFO - cn-jenkins-deploy-platform-ansible-os-6129-30:contrail-control module connection to 10.0.0.249:9041 servers UP
2026-08-03 05:12:58,728 - INFO - cn-jenkins-deploy-platform-ansible-os-6129-30:contrail-control module connection to 10.0.0.249:5673 servers UP
2026-08-03 05:12:58,728 - INFO - cn-jenkins-deploy-platform-ansible-os-6129-30:contrail-control module connection to ['10.0.0.249:8086'] servers UP
2026-08-03 05:12:58,728 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-6129-2.?flat
2026-08-03 05:12:58,759 - DEBUG - contrail-api is running
2026-08-03 05:12:58,760 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-6129-3.?flat
2026-08-03 05:12:58,791 - DEBUG - contrail-api is running
2026-08-03 05:12:58,791 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-6129-30.?flat
2026-08-03 05:12:58,816 - DEBUG - contrail-api is running
2026-08-03 05:12:59,235 - INFO - Created VN ctest-vn3-10370796, UUID :e9a33921-fe9a-45a7-b4f4-65f11134f1d6
2026-08-03 05:12:59,802 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2026-08-03 05:12:59,831 - DEBUG - Requesting: http://10.0.0.249:8082/domain/724de60e-9f87-4d23-867c-6bd9a9880f5b
2026-08-03 05:12:59,851 - DEBUG - Requesting: http://10.0.0.249:8082/project/34ffcb1a-09fd-4528-8d5b-a11d79da58f5
2026-08-03 05:12:59,957 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/037fe6af-1c0e-4e8c-98f0-f57c25b33b36
2026-08-03 05:12:59,978 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/037fe6af-1c0e-4e8c-98f0-f57c25b33b36
2026-08-03 05:12:59,994 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/dbe4fe15-976a-4496-9314-6c8fd9559164
2026-08-03 05:13:00,007 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/dbe4fe15-976a-4496-9314-6c8fd9559164
2026-08-03 05:13:00,019 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/c48e4bf4-4bf1-4397-8398-f7cb4cf592eb
2026-08-03 05:13:00,029 - DEBUG - Route Targets: ['target:64512:8000004']
2026-08-03 05:13:00,029 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/037fe6af-1c0e-4e8c-98f0-f57c25b33b36
2026-08-03 05:13:00,043 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/dbe4fe15-976a-4496-9314-6c8fd9559164
2026-08-03 05:13:00,068 - INFO - Verified VN network id 13 for VN 037fe6af-1c0e-4e8c-98f0-f57c25b33b36
2026-08-03 05:13:00,068 - INFO - Verifications in API Server for VN ctest-vn1-43687081 passed
2026-08-03 05:13:00,068 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/037fe6af-1c0e-4e8c-98f0-f57c25b33b36
2026-08-03 05:13:00,082 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/dbe4fe15-976a-4496-9314-6c8fd9559164
2026-08-03 05:13:00,094 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/dbe4fe15-976a-4496-9314-6c8fd9559164
2026-08-03 05:13:00,105 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/c48e4bf4-4bf1-4397-8398-f7cb4cf592eb
2026-08-03 05:13:00,121 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'interests': '1', 'advertised': '1', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-48746621', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '252173743920402060', 'uuid-lslong': '11020578201732856630'}, 'enable': 'true', 'created': '2026-08-03T05:09:46', 'last-modified': '2026-08-03T05:09:47', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:13.316296'}
2026-08-03 05:13:00,129 - DEBUG - Route Targets: ['target:64512:8000004']
2026-08-03 05:13:00,136 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-48746621', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '252173743920402060', 'uuid-lslong': '11020578201732856630'}, 'enable': 'true', 'created': '2026-08-03T05:09:46', 'last-modified': '2026-08-03T05:09:47', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:07.834837'}
2026-08-03 05:13:00,144 - DEBUG - Route Targets: ['target:64512:8000004']
2026-08-03 05:13:00,152 - DEBUG - Control-node 10.0.0.161 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-48746621', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '252173743920402060', 'uuid-lslong': '11020578201732856630'}, 'enable': 'true', 'created': '2026-08-03T05:09:46', 'last-modified': '2026-08-03T05:09:47', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:02.139950'}
2026-08-03 05:13:00,163 - DEBUG - Route Targets: ['target:64512:8000004']
2026-08-03 05:13:00,163 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-43687081 passed
2026-08-03 05:13:00,163 - DEBUG - ====Verifying policy data for ctest-vn1-43687081 in API_Server ======
2026-08-03 05:13:00,164 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2026-08-03 05:13:00,172 - DEBUG - Requesting: http://10.0.0.249:8082/domain/724de60e-9f87-4d23-867c-6bd9a9880f5b
2026-08-03 05:13:00,187 - DEBUG - Requesting: http://10.0.0.249:8082/project/34ffcb1a-09fd-4528-8d5b-a11d79da58f5
2026-08-03 05:13:00,310 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/037fe6af-1c0e-4e8c-98f0-f57c25b33b36
2026-08-03 05:13:00,327 - DEBUG - ==>Verifying data for policy with id: 2e88a82b-e858-40ba-b633-aac074b77ca9, fqn: ['default-domain', 'ctest-TestBasicVMVN0-48746621', 'policy1']
2026-08-03 05:13:00,327 - INFO - VN ctest-vn1-43687081 Policy verification: verify_vn_policy_in_api_server, status: True
2026-08-03 05:13:00,327 - DEBUG - Verifying the vn in opserver
2026-08-03 05:13:00,327 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 virtual network link through opserver 10.0.0.249
2026-08-03 05:13:00,328 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks
2026-08-03 05:13:00,346 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081?flat'}
2026-08-03 05:13:00,346 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 virtual network link through opserver 10.0.0.50
2026-08-03 05:13:00,346 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks
2026-08-03 05:13:00,402 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081?flat'}
2026-08-03 05:13:00,402 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 virtual network link through opserver 10.0.0.161
2026-08-03 05:13:00,402 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-networks
2026-08-03 05:13:00,491 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'href': 'http://10.0.0.161:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081?flat'}
2026-08-03 05:13:00,491 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is found in opserver
2026-08-03 05:13:00,507 - DEBUG - VRF ids for VN ctest-vn1-43687081: {'10.0.0.19': '2'}
2026-08-03 05:13:01,296 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-6129-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-6129-2)
2026-08-03 05:13:01,345 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:13:06,453 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:13:11,579 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:13:16,673 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:13:21,789 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:13:26,887 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:13:32,005 - DEBUG - VM is in ACTIVE state now
2026-08-03 05:13:32,006 - INFO - VM name : ctest-vn1_vm2-38063177
2026-08-03 05:13:32,122 - DEBUG - VM ctest-vn1_vm2-38063177 ID is 07750ad5-b5b5-43a8-90db-8df2e3967090
2026-08-03 05:13:32,222 - DEBUG - VM is in ACTIVE state now
2026-08-03 05:13:32,222 - DEBUG - VM ctest-vn1_vm2-38063177 launched on Node an-jenkins-deploy-platform-ansible-os-6129-2
2026-08-03 05:13:32,349 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090
2026-08-03 05:13:32,386 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090
2026-08-03 05:13:32,397 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/96157e08-4adb-471f-a875-63141c2cae23
2026-08-03 05:13:32,412 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090
2026-08-03 05:13:32,427 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/96157e08-4adb-471f-a875-63141c2cae23
2026-08-03 05:13:32,441 - DEBUG - Requesting: http://10.0.0.161:8082/instance-ip/951a7415-2a6b-4c6a-86d4-006e03a3d6bb
2026-08-03 05:13:32,455 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:13:32,455 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090
2026-08-03 05:13:32,468 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/96157e08-4adb-471f-a875-63141c2cae23
2026-08-03 05:13:32,501 - DEBUG - Requesting: http://10.0.0.161:8082/instance-ip/951a7415-2a6b-4c6a-86d4-006e03a3d6bb
2026-08-03 05:13:32,514 - INFO - VM ctest-vn1_vm2-38063177 verfication in all API Servers passed
2026-08-03 05:13:32,558 - DEBUG - VM ctest-vn1_vm2-38063177 Tap interface: {'index': '4', 'name': 'tap96157e08-4a', 'uuid': '96157e08-4adb-471f-a875-63141c2cae23', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081:ctest-vn1-43687081', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'vm_uuid': '07750ad5-b5b5-43a8-90db-8df2e3967090', 'vm_name': 'ctest-vn1_vm2-38063177', 'ip_addr': '192.168.1.251', 'mac_addr': '02:96:15:7e:08:4a', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '145', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-48746621:96157e08-4adb-471f-a875-63141c2cae23', 'sg_uuid_list': ['4360d165-c92f-4ce9-887c-ba23a9f642bf'], 'static_route_list': None, 'vm_project_uuid': '34ffcb1a-09fd-4528-8d5b-a11d79da58f5', 'admin_state': 'Enabled', 'flow_key_idx': '35', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2026-08-03 05:13:32,559 - DEBUG - Agent 10.0.0.19 vrf name: default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081:ctest-vn1-43687081
2026-08-03 05:13:32,593 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081:ctest-vn1-43687081', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.254:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]}
2026-08-03 05:13:32,703 - DEBUG - VM ctest-vn1_vm2-38063177 : Tap interface tap96157e08-4a is set to Active
2026-08-03 05:13:32,703 - DEBUG - Tap interface tap96157e08-4a detail : {'index': '4', 'name': 'tap96157e08-4a', 'uuid': '96157e08-4adb-471f-a875-63141c2cae23', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081:ctest-vn1-43687081', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'vm_uuid': '07750ad5-b5b5-43a8-90db-8df2e3967090', 'vm_name': 'ctest-vn1_vm2-38063177', 'ip_addr': '192.168.1.251', 'mac_addr': '02:96:15:7e:08:4a', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '145', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-48746621:96157e08-4adb-471f-a875-63141c2cae23', 'sg_uuid_list': ['4360d165-c92f-4ce9-887c-ba23a9f642bf'], 'static_route_list': None, 'vm_project_uuid': '34ffcb1a-09fd-4528-8d5b-a11d79da58f5', 'admin_state': 'Enabled', 'flow_key_idx': '35', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2026-08-03 05:13:32,830 - DEBUG - Starting Layer 2 verification in Agent
2026-08-03 05:13:32,846 - DEBUG - Layer 2 path is seen for VM MAC 02:96:15:7e:08:4a in agent 10.0.0.19
2026-08-03 05:13:32,846 - DEBUG - Active layer 2 route in agent is present for VMI tap96157e08-4a
2026-08-03 05:13:32,846 - DEBUG - L2 label(34) matches bw route table and interface table
2026-08-03 05:13:32,877 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is found in Agent of node 10.0.0.19
2026-08-03 05:13:33,000 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is consistent in agent 10.0.0.19
2026-08-03 05:13:33,000 - DEBUG - Route for VM IP ['192.168.1.251'] is consistent in agent 10.0.0.19
2026-08-03 05:13:33,000 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 verification for VM ctest-vn1_vm2-38063177 in Agent 10.0.0.19 passed
2026-08-03 05:13:33,123 - DEBUG - Starting all layer 2 verification in agent 10.0.0.19
2026-08-03 05:13:33,137 - DEBUG - Route for VM MAC 02:96:15:7e:08:4a is consistent in agent 10.0.0.19
2026-08-03 05:13:36,326 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1001ms')
2026-08-03 05:13:36,326 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-38063177 failed!
2026-08-03 05:13:36,345 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:13:40,424 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms')
2026-08-03 05:13:40,424 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-38063177 failed!
2026-08-03 05:13:40,443 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:13:44,515 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1008ms')
2026-08-03 05:13:44,516 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-38063177 failed!
2026-08-03 05:13:44,538 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:13:48,608 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms')
2026-08-03 05:13:48,609 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-38063177 failed!
2026-08-03 05:13:48,628 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:13:52,708 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms')
2026-08-03 05:13:52,708 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-38063177 failed!
2026-08-03 05:13:52,729 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:13:56,809 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1006ms')
2026-08-03 05:13:56,809 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-38063177 failed!
2026-08-03 05:13:56,830 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:14:00,903 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1008ms')
2026-08-03 05:14:00,903 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-38063177 failed!
2026-08-03 05:14:00,929 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:14:05,033 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1035ms')
2026-08-03 05:14:05,033 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-38063177 failed!
2026-08-03 05:14:05,057 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 is 192.168.1.254 and allocation pool is NOT set
2026-08-03 05:14:07,129 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=11.7 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=5.09 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 5.085/8.405/11.726/3.320 ms')
2026-08-03 05:14:07,129 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-38063177 passed
2026-08-03 05:14:07,129 - INFO - VM ctest-vn1_vm2-38063177 verifications in Compute nodes passed
2026-08-03 05:14:07,536 - DEBUG - Validated VM route 192.168.1.251 in vrouter of 10.0.0.19
2026-08-03 05:14:07,544 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0
2026-08-03 05:14:07,544 - INFO - Validated routes of VM ctest-vn1_vm2-38063177 in all vrouters
2026-08-03 05:14:08,007 - DEBUG - Starting all layer2 verification in 10.0.0.161 Control Node
2026-08-03 05:14:08,150 - DEBUG - Layer2 route found for VM MAC 02:96:15:7e:08:4a in Control-node 10.0.0.161
2026-08-03 05:14:08,150 - DEBUG - L2 Label for VM ctest-vn1_vm2-38063177 same between Control-node 10.0.0.161 and Agent, Expected: 34, Seen: 34
2026-08-03 05:14:08,150 - DEBUG - Starting all layer2 verification in 10.0.0.249 Control Node
2026-08-03 05:14:08,298 - DEBUG - Layer2 route found for VM MAC 02:96:15:7e:08:4a in Control-node 10.0.0.249
2026-08-03 05:14:08,299 - DEBUG - L2 Label for VM ctest-vn1_vm2-38063177 same between Control-node 10.0.0.249 and Agent, Expected: 34, Seen: 34
2026-08-03 05:14:08,299 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-38063177 passed
2026-08-03 05:14:08,299 - DEBUG - Verifying the vm in opserver
2026-08-03 05:14:08,299 - DEBUG - Verifying in collector 10.0.0.161 ...
2026-08-03 05:14:08,299 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090?flat
2026-08-03 05:14:08,369 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-48746621:96157e08-4adb-471f-a875-63141c2cae23?flat
2026-08-03 05:14:08,395 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-48746621:96157e08-4adb-471f-a875-63141c2cae23?flat
2026-08-03 05:14:08,431 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090?flat
2026-08-03 05:14:08,456 - DEBUG - Verifying vm in vn uve
2026-08-03 05:14:08,456 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-48746621:96157e08-4adb-471f-a875-63141c2cae23?flat
2026-08-03 05:14:08,489 - DEBUG - VM uve shows interface as {'vhostuser_mode': 0, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081', 'admin_state': True, 'fixed_ip4_list': ['192.168.1.251'], 'vn_uuid': '037fe6af-1c0e-4e8c-98f0-f57c25b33b36', 'vm_name': 'ctest-vn1_vm2-38063177', 'mac_address': '02:96:15:7e:08:4a', 'ip_address': '192.168.1.251', '__T': 1785734021253071, 'ip6_active': False, 'uuid': '96157e08-4adb-471f-a875-63141c2cae23', 'ip6_address': '::', 'tx_vlan': 65535, 'l2_active': True, 'rx_vlan': 65535, 'vm_uuid': '07750ad5-b5b5-43a8-90db-8df2e3967090', 'ip4_active': True, 'is_health_check_active': True, 'gateway': '192.168.1.254', 'active': True, 'port_mirror_enabled': False, 'label': 30}
2026-08-03 05:14:08,489 - DEBUG - VM uve shows ip address as ['192.168.1.251', '::']
2026-08-03 05:14:08,489 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081
2026-08-03 05:14:08,490 - INFO - Verifying through opserver in 10.0.0.249
2026-08-03 05:14:08,490 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081?flat
2026-08-03 05:14:08,521 - DEBUG - expected vm list 07750ad5-b5b5-43a8-90db-8df2e3967090
2026-08-03 05:14:08,521 - DEBUG - Extracted vm list ['07750ad5-b5b5-43a8-90db-8df2e3967090']
2026-08-03 05:14:08,521 - INFO - VM 07750ad5-b5b5-43a8-90db-8df2e3967090 is present in default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081
2026-08-03 05:14:08,523 - DEBUG - Verifying vm in vrouter uve
2026-08-03 05:14:08,523 - DEBUG - Getting info from collector 10.0.0.161..
2026-08-03 05:14:08,523 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090?flat
2026-08-03 05:14:08,598 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-6129-2']
2026-08-03 05:14:08,598 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-2:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:14:08,660 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-2?flat
2026-08-03 05:14:08,765 - DEBUG - VM 07750ad5-b5b5-43a8-90db-8df2e3967090 is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-2 uve
2026-08-03 05:14:08,765 - DEBUG - Validated that VM ctest-vn1_vm2-38063177 is in Vrouter an-jenkins-deploy-platform-ansible-os-6129-2 UVE
2026-08-03 05:14:08,765 - DEBUG - Verifying vm tap interface/vn in vrouter uve
2026-08-03 05:14:08,776 - DEBUG - Expected tap interface of VM uuid 07750ad5-b5b5-43a8-90db-8df2e3967090 is default-domain:ctest-TestBasicVMVN0-48746621:96157e08-4adb-471f-a875-63141c2cae23
2026-08-03 05:14:08,777 - DEBUG - Expected VN of VM uuid 07750ad5-b5b5-43a8-90db-8df2e3967090 is default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081
2026-08-03 05:14:08,777 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-2:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:14:08,861 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-2?flat
2026-08-03 05:14:08,945 - DEBUG - VM 07750ad5-b5b5-43a8-90db-8df2e3967090 is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-2 uve
2026-08-03 05:14:08,945 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-48746621:96157e08-4adb-471f-a875-63141c2cae23 of vm 07750ad5-b5b5-43a8-90db-8df2e3967090 is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-2 uve
2026-08-03 05:14:08,945 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 of vm 07750ad5-b5b5-43a8-90db-8df2e3967090 is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-2 uve
2026-08-03 05:14:08,945 - DEBUG - Verifying the 07750ad5-b5b5-43a8-90db-8df2e3967090 virtual network link through opserver 10.0.0.249
2026-08-03 05:14:08,945 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machines
2026-08-03 05:14:08,960 - DEBUG - VM link and name as {'name': '07750ad5-b5b5-43a8-90db-8df2e3967090', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090?flat'}
2026-08-03 05:14:08,960 - DEBUG - Verifying the 07750ad5-b5b5-43a8-90db-8df2e3967090 virtual network link through opserver 10.0.0.50
2026-08-03 05:14:08,960 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines
2026-08-03 05:14:08,974 - DEBUG - VM link and name as {'name': '07750ad5-b5b5-43a8-90db-8df2e3967090', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090?flat'}
2026-08-03 05:14:08,974 - DEBUG - Verifying the 07750ad5-b5b5-43a8-90db-8df2e3967090 virtual network link through opserver 10.0.0.161
2026-08-03 05:14:08,974 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machines
2026-08-03 05:14:08,999 - DEBUG - VM link and name as {'name': '07750ad5-b5b5-43a8-90db-8df2e3967090', 'href': 'http://10.0.0.161:8081/analytics/uves/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090?flat'}
2026-08-03 05:14:08,999 - INFO - VM ctest-vn1_vm2-38063177 validations in Opserver passed
2026-08-03 05:14:09,777 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-6129-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-6129-1)
2026-08-03 05:14:09,830 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:14:14,949 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:14:20,069 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:14:25,169 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:14:30,274 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:14:35,371 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2026-08-03 05:14:40,496 - DEBUG - VM is in ACTIVE state now
2026-08-03 05:14:40,496 - INFO - VM name : ctest-vn2_vm2-45396190
2026-08-03 05:14:40,614 - DEBUG - VM ctest-vn2_vm2-45396190 ID is 95e6831e-f935-49f2-b4fe-538c07b566f2
2026-08-03 05:14:40,722 - DEBUG - VM is in ACTIVE state now
2026-08-03 05:14:40,722 - DEBUG - VM ctest-vn2_vm2-45396190 launched on Node an-jenkins-deploy-platform-ansible-os-6129-1
2026-08-03 05:14:40,834 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2
2026-08-03 05:14:40,848 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2
2026-08-03 05:14:40,859 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/16c62997-7b7c-442d-9ebb-9bdd5aff0eb2
2026-08-03 05:14:40,873 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2
2026-08-03 05:14:40,887 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/16c62997-7b7c-442d-9ebb-9bdd5aff0eb2
2026-08-03 05:14:40,907 - DEBUG - Requesting: http://10.0.0.161:8082/instance-ip/2290bbaa-382a-4c07-ac2e-dec43b5fae48
2026-08-03 05:14:40,922 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:14:40,922 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2
2026-08-03 05:14:40,935 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/16c62997-7b7c-442d-9ebb-9bdd5aff0eb2
2026-08-03 05:14:40,953 - DEBUG - Requesting: http://10.0.0.161:8082/instance-ip/2290bbaa-382a-4c07-ac2e-dec43b5fae48
2026-08-03 05:14:40,969 - INFO - VM ctest-vn2_vm2-45396190 verfication in all API Servers passed
2026-08-03 05:14:40,999 - DEBUG - VM ctest-vn2_vm2-45396190 Tap interface: {'index': '4', 'name': 'tap16c62997-7b', 'uuid': '16c62997-7b7c-442d-9ebb-9bdd5aff0eb2', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240:ctest-vn2-81446240', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240', 'vm_uuid': '95e6831e-f935-49f2-b4fe-538c07b566f2', 'vm_name': 'ctest-vn2_vm2-45396190', 'ip_addr': '192.168.2.251', 'mac_addr': '02:16:c6:29:97:7b', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '101', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-48746621:16c62997-7b7c-442d-9ebb-9bdd5aff0eb2', 'sg_uuid_list': ['4360d165-c92f-4ce9-887c-ba23a9f642bf'], 'static_route_list': None, 'vm_project_uuid': '34ffcb1a-09fd-4528-8d5b-a11d79da58f5', 'admin_state': 'Enabled', 'flow_key_idx': '20', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2026-08-03 05:14:40,999 - DEBUG - Agent 10.0.0.38 vrf name: default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240:ctest-vn2-81446240
2026-08-03 05:14:41,008 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240:ctest-vn2-81446240', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240', 'table_label': '-1', 'vxlan_id': '14', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.31:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]}
2026-08-03 05:14:41,132 - DEBUG - VM ctest-vn2_vm2-45396190 : Tap interface tap16c62997-7b is set to Active
2026-08-03 05:14:41,132 - DEBUG - Tap interface tap16c62997-7b detail : {'index': '4', 'name': 'tap16c62997-7b', 'uuid': '16c62997-7b7c-442d-9ebb-9bdd5aff0eb2', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240:ctest-vn2-81446240', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240', 'vm_uuid': '95e6831e-f935-49f2-b4fe-538c07b566f2', 'vm_name': 'ctest-vn2_vm2-45396190', 'ip_addr': '192.168.2.251', 'mac_addr': '02:16:c6:29:97:7b', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '101', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-48746621:16c62997-7b7c-442d-9ebb-9bdd5aff0eb2', 'sg_uuid_list': ['4360d165-c92f-4ce9-887c-ba23a9f642bf'], 'static_route_list': None, 'vm_project_uuid': '34ffcb1a-09fd-4528-8d5b-a11d79da58f5', 'admin_state': 'Enabled', 'flow_key_idx': '20', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2026-08-03 05:14:41,260 - DEBUG - Starting Layer 2 verification in Agent
2026-08-03 05:14:41,285 - DEBUG - Layer 2 path is seen for VM MAC 02:16:c6:29:97:7b in agent 10.0.0.38
2026-08-03 05:14:41,286 - DEBUG - Active layer 2 route in agent is present for VMI tap16c62997-7b
2026-08-03 05:14:41,286 - DEBUG - L2 label(34) matches bw route table and interface table
2026-08-03 05:14:41,312 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 is found in Agent of node 10.0.0.38
2026-08-03 05:14:41,447 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 is consistent in agent 10.0.0.38
2026-08-03 05:14:41,447 - DEBUG - Route for VM IP ['192.168.2.251'] is consistent in agent 10.0.0.38
2026-08-03 05:14:41,447 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 verification for VM ctest-vn2_vm2-45396190 in Agent 10.0.0.38 passed
2026-08-03 05:14:41,574 - DEBUG - Starting all layer 2 verification in agent 10.0.0.38
2026-08-03 05:14:41,593 - DEBUG - Route for VM MAC 02:16:c6:29:97:7b is consistent in agent 10.0.0.38
2026-08-03 05:14:44,817 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms')
2026-08-03 05:14:44,817 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45396190 failed!
2026-08-03 05:14:44,836 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 is 192.168.2.254 and allocation pool is NOT set
2026-08-03 05:14:48,905 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1005ms')
2026-08-03 05:14:48,905 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45396190 failed!
2026-08-03 05:14:48,927 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 is 192.168.2.254 and allocation pool is NOT set
2026-08-03 05:14:53,009 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms')
2026-08-03 05:14:53,009 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45396190 failed!
2026-08-03 05:14:53,032 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 is 192.168.2.254 and allocation pool is NOT set
2026-08-03 05:14:57,103 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1002ms')
2026-08-03 05:14:57,103 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45396190 failed!
2026-08-03 05:14:57,123 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 is 192.168.2.254 and allocation pool is NOT set
2026-08-03 05:15:01,230 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1035ms')
2026-08-03 05:15:01,230 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45396190 failed!
2026-08-03 05:15:01,251 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 is 192.168.2.254 and allocation pool is NOT set
2026-08-03 05:15:05,324 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1008ms')
2026-08-03 05:15:05,325 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45396190 failed!
2026-08-03 05:15:05,347 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 is 192.168.2.254 and allocation pool is NOT set
2026-08-03 05:15:09,429 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms')
2026-08-03 05:15:09,429 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45396190 failed!
2026-08-03 05:15:09,449 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 is 192.168.2.254 and allocation pool is NOT set
2026-08-03 05:15:13,548 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1030ms')
2026-08-03 05:15:13,548 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45396190 failed!
2026-08-03 05:15:13,568 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 is 192.168.2.254 and allocation pool is NOT set
2026-08-03 05:15:17,649 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=8.31 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1011ms\r\nrtt min/avg/max/mdev = 8.309/8.309/8.309/0.000 ms')
2026-08-03 05:15:17,649 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45396190 passed
2026-08-03 05:15:17,649 - INFO - VM ctest-vn2_vm2-45396190 verifications in Compute nodes passed
2026-08-03 05:15:17,961 - DEBUG - Validated VM route 192.168.2.251 in vrouter of 10.0.0.38
2026-08-03 05:15:17,967 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0
2026-08-03 05:15:17,967 - INFO - Validated routes of VM ctest-vn2_vm2-45396190 in all vrouters
2026-08-03 05:15:18,444 - DEBUG - Starting all layer2 verification in 10.0.0.161 Control Node
2026-08-03 05:15:18,673 - DEBUG - Layer2 route found for VM MAC 02:16:c6:29:97:7b in Control-node 10.0.0.161
2026-08-03 05:15:18,673 - DEBUG - L2 Label for VM ctest-vn2_vm2-45396190 same between Control-node 10.0.0.161 and Agent, Expected: 34, Seen: 34
2026-08-03 05:15:18,673 - DEBUG - Starting all layer2 verification in 10.0.0.249 Control Node
2026-08-03 05:15:18,850 - DEBUG - Layer2 route found for VM MAC 02:16:c6:29:97:7b in Control-node 10.0.0.249
2026-08-03 05:15:18,850 - DEBUG - L2 Label for VM ctest-vn2_vm2-45396190 same between Control-node 10.0.0.249 and Agent, Expected: 34, Seen: 34
2026-08-03 05:15:18,850 - INFO - Verification in Control-nodes for VM ctest-vn2_vm2-45396190 passed
2026-08-03 05:15:18,851 - DEBUG - Verifying the vm in opserver
2026-08-03 05:15:18,851 - DEBUG - Verifying in collector 10.0.0.161 ...
2026-08-03 05:15:18,851 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2?flat
2026-08-03 05:15:18,883 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-48746621:16c62997-7b7c-442d-9ebb-9bdd5aff0eb2?flat
2026-08-03 05:15:18,910 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-48746621:16c62997-7b7c-442d-9ebb-9bdd5aff0eb2?flat
2026-08-03 05:15:18,937 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2?flat
2026-08-03 05:15:18,959 - DEBUG - Verifying vm in vn uve
2026-08-03 05:15:18,959 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-48746621:16c62997-7b7c-442d-9ebb-9bdd5aff0eb2?flat
2026-08-03 05:15:18,987 - DEBUG - VM uve shows interface as {'ip6_address': '::', 'fixed_ip4_list': ['192.168.2.251'], 'ip6_active': False, 'active': True, 'mac_address': '02:16:c6:29:97:7b', 'port_mirror_enabled': False, 'rx_vlan': 65535, 'uuid': '16c62997-7b7c-442d-9ebb-9bdd5aff0eb2', 'vm_name': 'ctest-vn2_vm2-45396190', 'ip4_active': True, 'admin_state': True, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240', 'vm_uuid': '95e6831e-f935-49f2-b4fe-538c07b566f2', 'is_health_check_active': True, 'ip_address': '192.168.2.251', 'vn_uuid': 'a7c4faf9-ef5f-4c0c-b87e-bbeec2a5ab08', 'tx_vlan': 65535, 'gateway': '192.168.2.254', 'label': 30, 'l2_active': True, 'vhostuser_mode': 0, '__T': 1785734102486023}
2026-08-03 05:15:18,987 - DEBUG - VM uve shows ip address as ['192.168.2.251', '::']
2026-08-03 05:15:18,987 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240
2026-08-03 05:15:18,987 - INFO - Verifying through opserver in 10.0.0.249
2026-08-03 05:15:18,987 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240?flat
2026-08-03 05:15:19,032 - DEBUG - expected vm list 95e6831e-f935-49f2-b4fe-538c07b566f2
2026-08-03 05:15:19,032 - DEBUG - Extracted vm list ['95e6831e-f935-49f2-b4fe-538c07b566f2']
2026-08-03 05:15:19,032 - INFO - VM 95e6831e-f935-49f2-b4fe-538c07b566f2 is present in default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240
2026-08-03 05:15:19,033 - DEBUG - Verifying vm in vrouter uve
2026-08-03 05:15:19,033 - DEBUG - Getting info from collector 10.0.0.161..
2026-08-03 05:15:19,033 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2?flat
2026-08-03 05:15:19,060 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-6129-1']
2026-08-03 05:15:19,060 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-1:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:15:19,158 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-1?flat
2026-08-03 05:15:19,202 - DEBUG - VM 95e6831e-f935-49f2-b4fe-538c07b566f2 is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-1 uve
2026-08-03 05:15:19,202 - DEBUG - Validated that VM ctest-vn2_vm2-45396190 is in Vrouter an-jenkins-deploy-platform-ansible-os-6129-1 UVE
2026-08-03 05:15:19,202 - DEBUG - Verifying vm tap interface/vn in vrouter uve
2026-08-03 05:15:19,216 - DEBUG - Expected tap interface of VM uuid 95e6831e-f935-49f2-b4fe-538c07b566f2 is default-domain:ctest-TestBasicVMVN0-48746621:16c62997-7b7c-442d-9ebb-9bdd5aff0eb2
2026-08-03 05:15:19,216 - DEBUG - Expected VN of VM uuid 95e6831e-f935-49f2-b4fe-538c07b566f2 is default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240
2026-08-03 05:15:19,216 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-1:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:15:19,315 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-1?flat
2026-08-03 05:15:19,359 - DEBUG - VM 95e6831e-f935-49f2-b4fe-538c07b566f2 is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-1 uve
2026-08-03 05:15:19,359 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-48746621:16c62997-7b7c-442d-9ebb-9bdd5aff0eb2 of vm 95e6831e-f935-49f2-b4fe-538c07b566f2 is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-1 uve
2026-08-03 05:15:19,359 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 of vm 95e6831e-f935-49f2-b4fe-538c07b566f2 is present in vrouter an-jenkins-deploy-platform-ansible-os-6129-1 uve
2026-08-03 05:15:19,359 - DEBUG - Verifying the 95e6831e-f935-49f2-b4fe-538c07b566f2 virtual network link through opserver 10.0.0.249
2026-08-03 05:15:19,359 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machines
2026-08-03 05:15:19,386 - DEBUG - VM link and name as {'name': '95e6831e-f935-49f2-b4fe-538c07b566f2', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2?flat'}
2026-08-03 05:15:19,386 - DEBUG - Verifying the 95e6831e-f935-49f2-b4fe-538c07b566f2 virtual network link through opserver 10.0.0.50
2026-08-03 05:15:19,386 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines
2026-08-03 05:15:19,402 - DEBUG - VM link and name as {'name': '95e6831e-f935-49f2-b4fe-538c07b566f2', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2?flat'}
2026-08-03 05:15:19,402 - DEBUG - Verifying the 95e6831e-f935-49f2-b4fe-538c07b566f2 virtual network link through opserver 10.0.0.161
2026-08-03 05:15:19,402 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machines
2026-08-03 05:15:19,420 - DEBUG - VM link and name as {'name': '95e6831e-f935-49f2-b4fe-538c07b566f2', 'href': 'http://10.0.0.161:8081/analytics/uves/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2?flat'}
2026-08-03 05:15:19,420 - INFO - VM ctest-vn2_vm2-45396190 validations in Opserver passed
2026-08-03 05:15:19,420 - INFO - Waiting for VM ctest-vn1_vm2-38063177 to be up..
2026-08-03 05:15:19,420 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:15:19,420 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090
2026-08-03 05:15:19,432 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/96157e08-4adb-471f-a875-63141c2cae23
2026-08-03 05:15:20,627 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.99 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.780 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.780/2.386/3.993/1.606 ms')
2026-08-03 05:15:20,627 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-38063177 passed
2026-08-03 05:15:20,713 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:15:20,713 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-38063177, IP 192.168.1.251, Port 22
2026-08-03 05:15:20,891 - DEBUG - VM ctest-vn1_vm2-38063177 is ready for SSH connections
2026-08-03 05:15:20,892 - INFO - Waiting for VM ctest-vn2_vm2-45396190 to be up..
2026-08-03 05:15:20,892 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:15:20,892 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2
2026-08-03 05:15:20,904 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine-interface/16c62997-7b7c-442d-9ebb-9bdd5aff0eb2
2026-08-03 05:15:22,147 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=2.29 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.862 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.862/1.578/2.294/0.716 ms')
2026-08-03 05:15:22,147 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-45396190 passed
2026-08-03 05:15:22,227 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:15:22,227 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-45396190, IP 192.168.2.251, Port 22
2026-08-03 05:15:22,297 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2026-08-03 05:15:22,443 - DEBUG - VM ctest-vn2_vm2-45396190 is NOT ready for SSH connections, VM status: ACTIVE
2026-08-03 05:15:27,444 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:15:27,444 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-45396190, IP 192.168.2.251, Port 22
2026-08-03 05:15:27,512 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2026-08-03 05:15:27,610 - DEBUG - VM ctest-vn2_vm2-45396190 is NOT ready for SSH connections, VM status: ACTIVE
2026-08-03 05:15:32,610 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:15:32,610 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-45396190, IP 192.168.2.251, Port 22
2026-08-03 05:15:32,789 - DEBUG - VM ctest-vn2_vm2-45396190 is ready for SSH connections
2026-08-03 05:15:32,789 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2026-08-03 05:15:32,789 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.251, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.19, gateway password: c0ntrail123
2026-08-03 05:15:32,789 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.251
2026-08-03 05:16:25,922 - DEBUG - PING 192.168.2.251 (192.168.2.251) 56(84) bytes of data.
64 bytes from 192.168.2.251: icmp_req=1 ttl=63 time=12.3 ms
64 bytes from 192.168.2.251: icmp_req=2 ttl=63 time=1.87 ms
64 bytes from 192.168.2.251: icmp_req=3 ttl=63 time=1.76 ms
--- 192.168.2.251 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2005ms
rtt min/avg/max/mdev = 1.765/5.343/12.388/4.981 ms
2026-08-03 05:16:25,923 - INFO - Ping to IP 192.168.2.251 from VM ctest-vn1_vm2-38063177 passed
2026-08-03 05:16:25,923 - INFO - Deleting VM ctest-vn2_vm2-45396190
2026-08-03 05:16:26,123 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:16:26,123 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2
2026-08-03 05:16:26,135 - DEBUG - VM ID 95e6831e-f935-49f2-b4fe-538c07b566f2 of VM ctest-vn2_vm2-45396190 is still found in API Server
2026-08-03 05:16:28,136 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:16:28,136 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2
2026-08-03 05:16:28,147 - DEBUG - VM ID 95e6831e-f935-49f2-b4fe-538c07b566f2 of VM ctest-vn2_vm2-45396190 is still found in API Server
2026-08-03 05:16:30,148 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:16:30,148 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2
2026-08-03 05:16:30,159 - DEBUG - Response Code: 404
2026-08-03 05:16:30,159 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2
2026-08-03 05:16:30,169 - DEBUG - Response Code: 404
2026-08-03 05:16:30,169 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/95e6831e-f935-49f2-b4fe-538c07b566f2
2026-08-03 05:16:30,177 - DEBUG - Response Code: 404
2026-08-03 05:16:30,177 - INFO - VM ctest-vn2_vm2-45396190 is fully removed in API-Server
2026-08-03 05:16:30,345 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found
2026-08-03 05:16:30,580 - DEBUG - VM route 192.168.2.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 of compute 10.0.0.38
2026-08-03 05:16:30,589 - DEBUG - Validated that vrouter 10.0.0.38 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240
2026-08-03 05:16:30,590 - DEBUG - Validated that vrouter 10.0.0.19 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240
2026-08-03 05:16:30,590 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240
2026-08-03 05:16:30,590 - INFO - VM ctest-vn2_vm2-45396190 is removed in Compute, and routes are removed in all compute nodes
2026-08-03 05:16:30,827 - INFO - Routes for VM ctest-vn2_vm2-45396190 is removed in all control-nodes
2026-08-03 05:16:30,827 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-1:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:16:30,895 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-1?flat
2026-08-03 05:16:30,980 - DEBUG - VM 95e6831e-f935-49f2-b4fe-538c07b566f2 is still present in vrouter an-jenkins-deploy-platform-ansible-os-6129-1 uve
2026-08-03 05:16:30,980 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2026-08-03 05:16:30,981 - DEBUG - interface for vm 95e6831e-f935-49f2-b4fe-538c07b566f2 deleted from vrouter uve ...
2026-08-03 05:16:30,981 - DEBUG - Deleted interface not in error interface list ...
2026-08-03 05:16:33,981 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-1:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:16:34,048 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-1?flat
2026-08-03 05:16:34,094 - DEBUG - VM 95e6831e-f935-49f2-b4fe-538c07b566f2 is not present in vrouter an-jenkins-deploy-platform-ansible-os-6129-1 uve
2026-08-03 05:16:34,094 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2026-08-03 05:16:34,094 - DEBUG - interface for vm 95e6831e-f935-49f2-b4fe-538c07b566f2 deleted from vrouter uve ...
2026-08-03 05:16:34,094 - DEBUG - Deleted interface not in error interface list ...
2026-08-03 05:16:34,094 - DEBUG - Validated that VM 95e6831e-f935-49f2-b4fe-538c07b566f2 is removed in Vrouter UVE
2026-08-03 05:16:34,094 - DEBUG - Verifying through opserver in 10.0.0.249
2026-08-03 05:16:34,095 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240?flat
2026-08-03 05:16:34,126 - DEBUG - 95e6831e-f935-49f2-b4fe-538c07b566f2 BM not in default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn2-81446240 uve
2026-08-03 05:16:34,126 - DEBUG - Validated that VM 95e6831e-f935-49f2-b4fe-538c07b566f2 is not present in the VN UVE
2026-08-03 05:16:34,126 - DEBUG - Verifying the 95e6831e-f935-49f2-b4fe-538c07b566f2 virtual network link through opserver 10.0.0.249
2026-08-03 05:16:34,126 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machines
2026-08-03 05:16:34,143 - INFO - 95e6831e-f935-49f2-b4fe-538c07b566f2 vm uve deleted from opserver
2026-08-03 05:16:34,143 - DEBUG - Verifying the 95e6831e-f935-49f2-b4fe-538c07b566f2 virtual network link through opserver 10.0.0.50
2026-08-03 05:16:34,143 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines
2026-08-03 05:16:34,159 - INFO - 95e6831e-f935-49f2-b4fe-538c07b566f2 vm uve deleted from opserver
2026-08-03 05:16:34,159 - DEBUG - Verifying the 95e6831e-f935-49f2-b4fe-538c07b566f2 virtual network link through opserver 10.0.0.161
2026-08-03 05:16:34,159 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machines
2026-08-03 05:16:34,176 - INFO - 95e6831e-f935-49f2-b4fe-538c07b566f2 vm uve deleted from opserver
2026-08-03 05:16:34,176 - INFO - Validated that VM 95e6831e-f935-49f2-b4fe-538c07b566f2 is removed from Opserver
2026-08-03 05:16:34,176 - INFO - Deleting VM ctest-vn1_vm2-38063177
2026-08-03 05:16:34,266 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:16:34,266 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090
2026-08-03 05:16:34,279 - DEBUG - VM ID 07750ad5-b5b5-43a8-90db-8df2e3967090 of VM ctest-vn1_vm2-38063177 is still found in API Server
2026-08-03 05:16:36,279 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:16:36,279 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090
2026-08-03 05:16:36,290 - DEBUG - VM ID 07750ad5-b5b5-43a8-90db-8df2e3967090 of VM ctest-vn1_vm2-38063177 is still found in API Server
2026-08-03 05:16:38,291 - DEBUG - Verifying in api server 10.0.0.161
2026-08-03 05:16:38,291 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090
2026-08-03 05:16:38,301 - DEBUG - Response Code: 404
2026-08-03 05:16:38,301 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090
2026-08-03 05:16:38,310 - DEBUG - Response Code: 404
2026-08-03 05:16:38,310 - DEBUG - Requesting: http://10.0.0.161:8082/virtual-machine/07750ad5-b5b5-43a8-90db-8df2e3967090
2026-08-03 05:16:38,319 - DEBUG - Response Code: 404
2026-08-03 05:16:38,319 - INFO - VM ctest-vn1_vm2-38063177 is fully removed in API-Server
2026-08-03 05:16:38,485 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found
2026-08-03 05:16:38,527 - DEBUG - Validated that vrouter 10.0.0.38 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081
2026-08-03 05:16:38,694 - DEBUG - VM route 192.168.1.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 of compute 10.0.0.19
2026-08-03 05:16:38,701 - DEBUG - Validated that vrouter 10.0.0.19 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081
2026-08-03 05:16:38,701 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081
2026-08-03 05:16:38,701 - INFO - VM ctest-vn1_vm2-38063177 is removed in Compute, and routes are removed in all compute nodes
2026-08-03 05:16:38,932 - INFO - Routes for VM ctest-vn1_vm2-38063177 is removed in all control-nodes
2026-08-03 05:16:38,933 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-2:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:16:38,992 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-2?flat
2026-08-03 05:16:39,029 - DEBUG - VM 07750ad5-b5b5-43a8-90db-8df2e3967090 is still present in vrouter an-jenkins-deploy-platform-ansible-os-6129-2 uve
2026-08-03 05:16:39,029 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2026-08-03 05:16:39,029 - DEBUG - interface for vm 07750ad5-b5b5-43a8-90db-8df2e3967090 deleted from vrouter uve ...
2026-08-03 05:16:39,029 - DEBUG - Deleted interface not in error interface list ...
2026-08-03 05:16:42,030 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-6129-2:Compute:contrail-vrouter-agent:0?flat
2026-08-03 05:16:42,116 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-6129-2?flat
2026-08-03 05:16:42,156 - DEBUG - VM 07750ad5-b5b5-43a8-90db-8df2e3967090 is not present in vrouter an-jenkins-deploy-platform-ansible-os-6129-2 uve
2026-08-03 05:16:42,156 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2026-08-03 05:16:42,156 - DEBUG - interface for vm 07750ad5-b5b5-43a8-90db-8df2e3967090 deleted from vrouter uve ...
2026-08-03 05:16:42,156 - DEBUG - Deleted interface not in error interface list ...
2026-08-03 05:16:42,156 - DEBUG - Validated that VM 07750ad5-b5b5-43a8-90db-8df2e3967090 is removed in Vrouter UVE
2026-08-03 05:16:42,156 - DEBUG - Verifying through opserver in 10.0.0.249
2026-08-03 05:16:42,156 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081?flat
2026-08-03 05:16:42,191 - DEBUG - 07750ad5-b5b5-43a8-90db-8df2e3967090 BM not in default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081 uve
2026-08-03 05:16:42,191 - DEBUG - Validated that VM 07750ad5-b5b5-43a8-90db-8df2e3967090 is not present in the VN UVE
2026-08-03 05:16:42,191 - DEBUG - Verifying the 07750ad5-b5b5-43a8-90db-8df2e3967090 virtual network link through opserver 10.0.0.249
2026-08-03 05:16:42,191 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-machines
2026-08-03 05:16:42,210 - INFO - 07750ad5-b5b5-43a8-90db-8df2e3967090 vm uve deleted from opserver
2026-08-03 05:16:42,211 - DEBUG - Verifying the 07750ad5-b5b5-43a8-90db-8df2e3967090 virtual network link through opserver 10.0.0.50
2026-08-03 05:16:42,211 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-machines
2026-08-03 05:16:42,229 - INFO - 07750ad5-b5b5-43a8-90db-8df2e3967090 vm uve deleted from opserver
2026-08-03 05:16:42,229 - DEBUG - Verifying the 07750ad5-b5b5-43a8-90db-8df2e3967090 virtual network link through opserver 10.0.0.161
2026-08-03 05:16:42,229 - DEBUG - Requesting: http://10.0.0.161:8081/analytics/uves/virtual-machines
2026-08-03 05:16:42,248 - INFO - 07750ad5-b5b5-43a8-90db-8df2e3967090 vm uve deleted from opserver
2026-08-03 05:16:42,248 - INFO - Validated that VM 07750ad5-b5b5-43a8-90db-8df2e3967090 is removed from Opserver
2026-08-03 05:16:42,248 - INFO - Deleting VN ctest-vn3-10370796
2026-08-03 05:16:42,476 - INFO - Deleting VM ctest-vn2_vm1-81867373
2026-08-03 05:16:42,618 - INFO - Deleting VM ctest-vn1_vm1-46511698
2026-08-03 05:16:42,984 - INFO - Deleting VN ctest-vn2-81446240
2026-08-03 05:16:43,063 - DEBUG - RefsExistError Delete when resource still referred: ['http://10.0.0.161:8082/virtual-machine-interface/3696ad97-90fa-4d83-8d2b-361dd76c08ef'] while deleting VN ctest-vn2-81446240..Will retry
2026-08-03 05:16:45,383 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/7b24b36a-bcc8-4688-a72a-458ce9557d43
2026-08-03 05:16:45,394 - DEBUG - Response Code: 404
2026-08-03 05:16:45,395 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2026-08-03 05:16:45,402 - DEBUG - Requesting: http://10.0.0.249:8082/domain/724de60e-9f87-4d23-867c-6bd9a9880f5b
2026-08-03 05:16:45,415 - DEBUG - Requesting: http://10.0.0.249:8082/project/34ffcb1a-09fd-4528-8d5b-a11d79da58f5
2026-08-03 05:16:45,522 - INFO - Validated that VN ctest-vn2-81446240 is not found in API Server
2026-08-03 05:16:45,533 - DEBUG - VN ctest-vn2-81446240 is not present in Agent 10.0.0.38
2026-08-03 05:16:45,544 - DEBUG - VN ctest-vn2-81446240 is not present in Agent 10.0.0.19
2026-08-03 05:16:45,544 - INFO - Validated that VN ctest-vn2-81446240 is not in any agent
2026-08-03 05:16:45,584 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-81446240 info
2026-08-03 05:16:45,783 - INFO - Deleting VN ctest-vn1-43687081
2026-08-03 05:16:45,967 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/dbe4fe15-976a-4496-9314-6c8fd9559164
2026-08-03 05:16:45,973 - DEBUG - Response Code: 404
2026-08-03 05:16:45,973 - DEBUG - Requesting: http://10.0.0.249:8082/domains
2026-08-03 05:16:45,980 - DEBUG - Requesting: http://10.0.0.249:8082/domain/724de60e-9f87-4d23-867c-6bd9a9880f5b
2026-08-03 05:16:46,000 - DEBUG - Requesting: http://10.0.0.249:8082/project/34ffcb1a-09fd-4528-8d5b-a11d79da58f5
2026-08-03 05:16:46,121 - INFO - Validated that VN ctest-vn1-43687081 is not found in API Server
2026-08-03 05:16:46,140 - DEBUG - VN ctest-vn1-43687081 is not present in Agent 10.0.0.38
2026-08-03 05:16:46,155 - DEBUG - VN ctest-vn1-43687081 is not present in Agent 10.0.0.19
2026-08-03 05:16:46,155 - INFO - Validated that VN ctest-vn1-43687081 is not in any agent
2026-08-03 05:16:46,164 - DEBUG - VRF 2 is not seen in agent 10.0.0.19
2026-08-03 05:16:46,172 - DEBUG - Vrouter 10.0.0.19 does not have vrf 2 for VN ctest-vn1-43687081
2026-08-03 05:16:46,172 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestBasicVMVN0-48746621:ctest-vn1-43687081
2026-08-03 05:16:46,203 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-43687081 info
2026-08-03 05:16:46,269 - INFO - Deleted policy policy2
2026-08-03 05:16:46,345 - INFO - Deleted policy policy1
2026-08-03 05:16:47,714 - DEBUG - Skipping xmpp flap check
2026-08-03 05:16:47,715 - INFO - END TEST : test_process_restart_in_policy_between_vns : PASSED[0:07:04]
2026-08-03 05:16:47,715 - INFO - --------------------------------------------------------------------------------
2026-08-03 05:16:47,718 - INFO - ================================================================================
2026-08-03 05:16:47,718 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling
2026-08-03 05:16:47,718 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229).
1. Send broadcast traffic from one compute node.
2. Other compute in same subnet should receive that traffic.
3. Receiving compute should treat this traffic as underlay.
4. Compute should not replicate the packet and send the copy back.
Pass criteria: Step 3-4 should pass
Maintainer : chhandak@juniper.net
2026-08-03 05:16:49,067 - DEBUG - Skipping xmpp flap check
2026-08-03 05:16:49,067 - INFO - Initial checks done. Running the testcase now
2026-08-03 05:16:49,067 - INFO -
2026-08-03 05:16:49,501 - INFO - Agent interface name: enp5s0
2026-08-03 05:16:49,583 - DEBUG - Executing command: sudo tcpdump -nni enp5s0 -U host 10.20.0.255 -w /tmp/enp5s0_ctest-random-14562377.pcap
2026-08-03 05:16:55,081 - INFO - Pinging broacast address 10.20.0.255 from compute 10.20.0.31
2026-08-03 05:17:09,376 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}')
2026-08-03 05:17:11,529 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/enp5s0_ctest-random-14562377.pcap | grep -c length
2026-08-03 05:17:11,550 - DEBUG - STDOUT: 10
2026-08-03 05:17:11,550 - DEBUG - STDERR: reading from file /tmp/enp5s0_ctest-random-14562377.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2026-08-03 05:17:11,550 - INFO - 10 packets are found in tcpdump output as expected
2026-08-03 05:17:11,551 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}')
2026-08-03 05:17:13,593 - INFO - Packet count matched: Compute 10.0.0.19 has receive only 10 packet from source IP 10.20.0.255. No duplicate packet seen
2026-08-03 05:17:14,926 - DEBUG - Skipping xmpp flap check
2026-08-03 05:17:14,927 - INFO - END TEST : test_underlay_broadcast_traffic_handling : PASSED[0:00:27]
2026-08-03 05:17:14,927 - INFO - --------------------------------------------------------------------------------
2026-08-03 05:17:15,902 - INFO - Deleted project: ctest-TestBasicVMVN0-48746621, ID : 34ffcb1a-09fd-4528-8d5b-a11d79da58f5