2025-12-08 04:10:59,559 - INFO - Domain Default found not creating
2025-12-08 04:10:59,731 - INFO - Project ctest-TestBasicVMVN0-32026406 not found, creating it
2025-12-08 04:11:00,259 - INFO - Created Project:ctest-TestBasicVMVN0-32026406, ID : 39b9363a-5400-4ea1-a180-7dd8e2fe11b9
2025-12-08 04:11:02,172 - INFO - ================================================================================
2025-12-08 04:11:02,172 - INFO - STARTING TEST : test_control_node_switchover
2025-12-08 04:11:02,172 - INFO - TEST DESCRIPTION : Stop the control node and check peering with agent fallback to other control node.
1. Pick one VN from respource pool which has 2 VM's in it
2. Verify ping between VM's
3. Find active control node in cluster by agent inspect
4. Stop control service on active control node
5. Verify agents are connected to new active control-node using xmpp connections
6. Bring back control service on previous active node
7. Verify ping between VM's again after bringing up control serveice
Pass criteria: Step 2,5 and 7 should pass
2025-12-08 04:11:03,455 - DEBUG - Skipping xmpp flap check
2025-12-08 04:11:03,455 - INFO - Initial checks done. Running the testcase now
2025-12-08 04:11:03,455 - INFO -
2025-12-08 04:11:04,164 - DEBUG - Response for create_network : {'network': {'id': '3408134f-637d-48e3-9aee-4cb42988d140', 'name': 'ctest-vn1-49761356', 'tenant_id': '39b9363a54004ea1a1807dd8e2fe11b9', 'project_id': '39b9363a54004ea1a1807dd8e2fe11b9', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicVMVN0-32026406', 'ctest-vn1-49761356'], 'port_security_enabled': True, 'description': ''}}
2025-12-08 04:11:04,443 - DEBUG - Response for create_subnet : {'subnet': {'id': '0d6e3e24-2c4b-4e3b-b39e-e502588b2721', 'name': '', 'tenant_id': '39b9363a54004ea1a1807dd8e2fe11b9', 'network_id': '3408134f-637d-48e3-9aee-4cb42988d140', 'ip_version': 4, 'cidr': '192.168.1.0/24', 'allocation_pools': [{'start': '192.168.1.2', 'end': '192.168.1.254'}], 'gateway_ip': '192.168.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '192.168.1.2', 'tags': [], 'project_id': '39b9363a54004ea1a1807dd8e2fe11b9'}}
2025-12-08 04:11:04,466 - INFO - Created VN ctest-vn1-49761356
2025-12-08 04:11:04,483 - DEBUG - VN ctest-vn1-49761356 UUID is 3408134f-637d-48e3-9aee-4cb42988d140
2025-12-08 04:11:04,561 - DEBUG - Requesting: http://10.0.0.23:8082/domains
2025-12-08 04:11:04,910 - DEBUG - Requesting: http://10.0.0.23:8082/domains
2025-12-08 04:11:04,963 - DEBUG - Requesting: http://10.0.0.23:8082/domain/c5befdc1-279e-4ac1-9620-9608231491f8
2025-12-08 04:11:04,983 - DEBUG - Requesting: http://10.0.0.23:8082/project/39b9363a-5400-4ea1-a180-7dd8e2fe11b9
2025-12-08 04:11:05,073 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/3408134f-637d-48e3-9aee-4cb42988d140
2025-12-08 04:11:05,089 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/3408134f-637d-48e3-9aee-4cb42988d140
2025-12-08 04:11:05,100 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/e46942f0-8bc0-4b6a-951f-2a8248e1fbeb
2025-12-08 04:11:05,108 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/e46942f0-8bc0-4b6a-951f-2a8248e1fbeb
2025-12-08 04:11:05,116 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/354d0df7-3ac7-4e6f-a867-8eb6066d2614
2025-12-08 04:11:05,123 - DEBUG - Route Targets: ['target:64512:8000004']
2025-12-08 04:11:05,123 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/3408134f-637d-48e3-9aee-4cb42988d140
2025-12-08 04:11:05,132 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/e46942f0-8bc0-4b6a-951f-2a8248e1fbeb
2025-12-08 04:11:05,148 - INFO - Verified VN network id 14 for VN 3408134f-637d-48e3-9aee-4cb42988d140
2025-12-08 04:11:05,148 - INFO - Verifications in API Server for VN ctest-vn1-49761356 passed
2025-12-08 04:11:05,148 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/3408134f-637d-48e3-9aee-4cb42988d140
2025-12-08 04:11:05,158 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/e46942f0-8bc0-4b6a-951f-2a8248e1fbeb
2025-12-08 04:11:05,166 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/e46942f0-8bc0-4b6a-951f-2a8248e1fbeb
2025-12-08 04:11:05,173 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/354d0df7-3ac7-4e6f-a867-8eb6066d2614
2025-12-08 04:11:05,186 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-32026406', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3749267921478437091', 'uuid-lslong': '11163944862972694848'}, 'enable': 'true', 'created': '2025-12-08T04:11:04', 'last-modified': '2025-12-08T04:11:04', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.817145'}
2025-12-08 04:11:05,192 - DEBUG - Route Targets: ['target:64512:8000004']
2025-12-08 04:11:05,197 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-32026406', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3749267921478437091', 'uuid-lslong': '11163944862972694848'}, 'enable': 'true', 'created': '2025-12-08T04:11:04', 'last-modified': '2025-12-08T04:11:04', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.823421'}
2025-12-08 04:11:05,205 - DEBUG - Route Targets: ['target:64512:8000004']
2025-12-08 04:11:05,210 - DEBUG - Control-node 10.0.0.65 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-32026406', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3749267921478437091', 'uuid-lslong': '11163944862972694848'}, 'enable': 'true', 'created': '2025-12-08T04:11:04', 'last-modified': '2025-12-08T04:11:04', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.841046'}
2025-12-08 04:11:05,217 - DEBUG - Route Targets: ['target:64512:8000004']
2025-12-08 04:11:05,217 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-49761356 passed
2025-12-08 04:11:05,217 - DEBUG - ====Verifying policy data for ctest-vn1-49761356 in API_Server ======
2025-12-08 04:11:05,217 - DEBUG - Requesting: http://10.0.0.23:8082/domains
2025-12-08 04:11:05,223 - DEBUG - Requesting: http://10.0.0.23:8082/domain/c5befdc1-279e-4ac1-9620-9608231491f8
2025-12-08 04:11:05,236 - DEBUG - Requesting: http://10.0.0.23:8082/project/39b9363a-5400-4ea1-a180-7dd8e2fe11b9
2025-12-08 04:11:05,322 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/3408134f-637d-48e3-9aee-4cb42988d140
2025-12-08 04:11:05,338 - DEBUG - =>VN ctest-vn1-49761356 has no policy to be verified
2025-12-08 04:11:05,338 - DEBUG - Verifying the vn in opserver
2025-12-08 04:11:05,339 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 virtual network link through opserver 10.0.0.23
2025-12-08 04:11:05,339 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks
2025-12-08 04:11:05,711 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks
2025-12-08 04:11:05,778 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356?flat'}
2025-12-08 04:11:05,778 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 virtual network link through opserver 10.0.0.22
2025-12-08 04:11:05,779 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks
2025-12-08 04:11:06,144 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks
2025-12-08 04:11:06,217 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356?flat'}
2025-12-08 04:11:06,217 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 virtual network link through opserver 10.0.0.65
2025-12-08 04:11:06,217 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-networks
2025-12-08 04:11:06,564 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-networks
2025-12-08 04:11:06,633 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356', 'href': 'http://10.0.0.65:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356?flat'}
2025-12-08 04:11:06,633 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is found in opserver
2025-12-08 04:11:06,657 - DEBUG - Do not have enough data to verify VN in agent
2025-12-08 04:11:06,673 - DEBUG - VRF ids for VN ctest-vn1-49761356: {}
2025-12-08 04:11:06,738 - DEBUG - Services list from nova: [, , , ]
2025-12-08 04:11:07,943 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4854-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4854-2)
2025-12-08 04:11:09,516 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4854-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4854-1)
2025-12-08 04:11:09,517 - INFO - Waiting for VM ctest-vn1_vm1-39168219 to be up..
2025-12-08 04:11:09,609 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2025-12-08 04:11:14,700 - DEBUG - VM is in ACTIVE state now
2025-12-08 04:11:14,700 - INFO - VM name : ctest-vn1_vm1-39168219
2025-12-08 04:11:14,785 - DEBUG - VM ctest-vn1_vm1-39168219 ID is a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a
2025-12-08 04:11:14,812 - DEBUG - VM ctest-vn1_vm1-39168219 launched on Node an-jenkins-deploy-platform-ansible-os-4854-2
2025-12-08 04:11:14,892 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a
2025-12-08 04:11:15,228 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a
2025-12-08 04:11:15,265 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/f42d695c-be91-4a87-9a71-288bd82ffd95
2025-12-08 04:11:18,460 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1021ms')
2025-12-08 04:11:18,460 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-39168219 failed!
2025-12-08 04:11:18,474 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is 192.168.1.1 and allocation pool is NOT set
2025-12-08 04:11:22,561 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1025ms')
2025-12-08 04:11:22,561 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-39168219 failed!
2025-12-08 04:11:22,581 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is 192.168.1.1 and allocation pool is NOT set
2025-12-08 04:11:26,659 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms')
2025-12-08 04:11:26,659 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-39168219 failed!
2025-12-08 04:11:26,673 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is 192.168.1.1 and allocation pool is NOT set
2025-12-08 04:11:30,751 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms')
2025-12-08 04:11:30,751 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-39168219 failed!
2025-12-08 04:11:30,764 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is 192.168.1.1 and allocation pool is NOT set
2025-12-08 04:11:34,843 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms')
2025-12-08 04:11:34,843 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-39168219 failed!
2025-12-08 04:11:34,858 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is 192.168.1.1 and allocation pool is NOT set
2025-12-08 04:11:38,940 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms')
2025-12-08 04:11:38,940 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-39168219 failed!
2025-12-08 04:11:38,954 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is 192.168.1.1 and allocation pool is NOT set
2025-12-08 04:11:43,041 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms')
2025-12-08 04:11:43,041 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-39168219 failed!
2025-12-08 04:11:43,057 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is 192.168.1.1 and allocation pool is NOT set
2025-12-08 04:11:47,139 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms')
2025-12-08 04:11:47,139 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-39168219 failed!
2025-12-08 04:11:47,153 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is 192.168.1.1 and allocation pool is NOT set
2025-12-08 04:11:51,232 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms')
2025-12-08 04:11:51,232 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-39168219 failed!
2025-12-08 04:11:51,245 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is 192.168.1.1 and allocation pool is NOT set
2025-12-08 04:11:53,312 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=6.02 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.49 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 1.485/3.752/6.019/2.267 ms')
2025-12-08 04:11:53,312 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-39168219 passed
2025-12-08 04:11:53,383 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:11:53,383 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-39168219, IP 192.168.1.3, Port 22
2025-12-08 04:11:53,450 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-12-08 04:11:53,584 - DEBUG - VM ctest-vn1_vm1-39168219 is NOT ready for SSH connections, VM status: ACTIVE
2025-12-08 04:11:58,585 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:11:58,585 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-39168219, IP 192.168.1.3, Port 22
2025-12-08 04:11:58,654 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-12-08 04:11:58,749 - DEBUG - VM ctest-vn1_vm1-39168219 is NOT ready for SSH connections, VM status: ACTIVE
2025-12-08 04:12:03,750 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:12:03,750 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-39168219, IP 192.168.1.3, Port 22
2025-12-08 04:12:03,817 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-12-08 04:12:03,917 - DEBUG - VM ctest-vn1_vm1-39168219 is NOT ready for SSH connections, VM status: ACTIVE
2025-12-08 04:12:08,917 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:12:08,918 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-39168219, IP 192.168.1.3, Port 22
2025-12-08 04:12:08,986 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-12-08 04:12:09,079 - DEBUG - VM ctest-vn1_vm1-39168219 is NOT ready for SSH connections, VM status: ACTIVE
2025-12-08 04:12:14,080 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:12:14,080 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-39168219, IP 192.168.1.3, Port 22
2025-12-08 04:12:14,245 - DEBUG - VM ctest-vn1_vm1-39168219 is ready for SSH connections
2025-12-08 04:12:14,246 - INFO - Waiting for VM ctest-vn1_vm2-03488037 to be up..
2025-12-08 04:12:14,339 - DEBUG - VM is in ACTIVE state now
2025-12-08 04:12:14,339 - INFO - VM name : ctest-vn1_vm2-03488037
2025-12-08 04:12:14,418 - DEBUG - VM ctest-vn1_vm2-03488037 ID is 959d5c74-59a2-45f1-8a5c-1b768b83341b
2025-12-08 04:12:14,419 - DEBUG - VM ctest-vn1_vm2-03488037 launched on Node an-jenkins-deploy-platform-ansible-os-4854-1
2025-12-08 04:12:14,497 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/959d5c74-59a2-45f1-8a5c-1b768b83341b
2025-12-08 04:12:14,507 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/acc70189-933a-4697-9162-a8a2c71a9484
2025-12-08 04:12:15,670 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.80 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=1.19 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 1.192/3.495/5.798/2.303 ms')
2025-12-08 04:12:15,671 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-03488037 passed
2025-12-08 04:12:15,738 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:12:15,738 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-03488037, IP 192.168.1.4, Port 22
2025-12-08 04:12:15,910 - DEBUG - VM ctest-vn1_vm2-03488037 is ready for SSH connections
2025-12-08 04:12:15,910 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:12:15,910 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.129, gateway password: c0ntrail123
2025-12-08 04:12:15,911 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4
2025-12-08 04:12:39,961 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=9.65 ms
64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.48 ms
64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.15 ms
--- 192.168.1.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.152/4.097/9.656/3.933 ms
2025-12-08 04:12:39,961 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-39168219 passed
2025-12-08 04:12:39,961 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:12:39,961 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.41, gateway password: c0ntrail123
2025-12-08 04:12:39,961 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3
2025-12-08 04:13:20,846 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=6.47 ms
64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.26 ms
64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.26 ms
--- 192.168.1.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.266/3.001/6.470/2.452 ms
2025-12-08 04:13:20,846 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-03488037 passed
2025-12-08 04:13:20,854 - INFO - Active control node from the Agent 10.0.0.129 is 10.0.0.22
2025-12-08 04:13:20,855 - INFO - Stoping the Control service in 10.0.0.22
2025-12-08 04:13:21,268 - INFO - 10.0.0.22
2025-12-08 04:13:21,268 - INFO - control:inactive
2025-12-08 04:13:21,268 - INFO - Contrail services control are down on nodes 10.0.0.22
2025-12-08 04:13:26,276 - INFO - Active control node from the Agent 10.0.0.129 is 10.0.0.23
2025-12-08 04:13:26,276 - INFO - Starting the Control service in 10.0.0.22
2025-12-08 04:13:26,625 - INFO - 10.0.0.22
2025-12-08 04:13:26,629 - INFO - control:initializing
2025-12-08 04:13:26,629 - DEBUG - defaultdict(, {'10.0.0.22': {'control': {'status': 'initializing', 'description': None}}})
2025-12-08 04:13:26,629 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-12-08 04:13:31,731 - INFO - 10.0.0.22
2025-12-08 04:13:31,737 - INFO - control:active
2025-12-08 04:13:31,737 - INFO - Contrail services ['control'] are up on nodes 10.0.0.22
2025-12-08 04:13:36,833 - DEBUG - VM is in ACTIVE state now
2025-12-08 04:13:36,834 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:13:36,834 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a
2025-12-08 04:13:36,842 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/f42d695c-be91-4a87-9a71-288bd82ffd95
2025-12-08 04:13:36,851 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a
2025-12-08 04:13:36,858 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/f42d695c-be91-4a87-9a71-288bd82ffd95
2025-12-08 04:13:36,869 - DEBUG - Requesting: http://10.0.0.65:8082/instance-ip/965949af-744f-4d9a-8d80-1d1203ce4c6e
2025-12-08 04:13:36,879 - DEBUG - Verifying in api server 10.0.0.65
2025-12-08 04:13:36,879 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a
2025-12-08 04:13:36,887 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/f42d695c-be91-4a87-9a71-288bd82ffd95
2025-12-08 04:13:36,897 - DEBUG - Requesting: http://10.0.0.65:8082/instance-ip/965949af-744f-4d9a-8d80-1d1203ce4c6e
2025-12-08 04:13:36,906 - INFO - VM ctest-vn1_vm1-39168219 verfication in all API Servers passed
2025-12-08 04:13:36,926 - DEBUG - VM ctest-vn1_vm1-39168219 Tap interface: {'index': '3', 'name': 'tapf42d695c-be', 'uuid': 'f42d695c-be91-4a87-9a71-288bd82ffd95', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356:ctest-vn1-49761356', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356', 'vm_uuid': 'a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a', 'vm_name': 'ctest-vn1_vm1-39168219', 'ip_addr': '192.168.1.3', 'mac_addr': '02:f4:2d:69:5c:be', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-32026406:f42d695c-be91-4a87-9a71-288bd82ffd95', 'sg_uuid_list': ['a08ad7f7-4fb9-4642-b81a-e5edfb06e45f'], 'static_route_list': None, 'vm_project_uuid': '39b9363a-5400-4ea1-a180-7dd8e2fe11b9', 'admin_state': 'Enabled', 'flow_key_idx': '25', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-12-08 04:13:36,926 - DEBUG - Agent 10.0.0.129 vrf name: default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356:ctest-vn1-49761356
2025-12-08 04:13:36,932 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356:ctest-vn1-49761356', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356', 'table_label': '-1', 'vxlan_id': '14', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.19:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]}
2025-12-08 04:13:37,012 - DEBUG - VM ctest-vn1_vm1-39168219 : Tap interface tapf42d695c-be is set to Active
2025-12-08 04:13:37,013 - DEBUG - Tap interface tapf42d695c-be detail : {'index': '3', 'name': 'tapf42d695c-be', 'uuid': 'f42d695c-be91-4a87-9a71-288bd82ffd95', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356:ctest-vn1-49761356', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356', 'vm_uuid': 'a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a', 'vm_name': 'ctest-vn1_vm1-39168219', 'ip_addr': '192.168.1.3', 'mac_addr': '02:f4:2d:69:5c:be', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '165', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-32026406:f42d695c-be91-4a87-9a71-288bd82ffd95', 'sg_uuid_list': ['a08ad7f7-4fb9-4642-b81a-e5edfb06e45f'], 'static_route_list': None, 'vm_project_uuid': '39b9363a-5400-4ea1-a180-7dd8e2fe11b9', 'admin_state': 'Enabled', 'flow_key_idx': '25', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-12-08 04:13:37,126 - DEBUG - Starting Layer 2 verification in Agent
2025-12-08 04:13:37,139 - DEBUG - Layer 2 path is seen for VM MAC 02:f4:2d:69:5c:be in agent 10.0.0.129
2025-12-08 04:13:37,139 - DEBUG - Active layer 2 route in agent is present for VMI tapf42d695c-be
2025-12-08 04:13:37,139 - DEBUG - L2 label(29) matches bw route table and interface table
2025-12-08 04:13:37,144 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is found in Agent of node 10.0.0.41
2025-12-08 04:13:37,275 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is consistent in agent 10.0.0.41
2025-12-08 04:13:37,275 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.41
2025-12-08 04:13:37,275 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 verification for VM ctest-vn1_vm1-39168219 in Agent 10.0.0.41 passed
2025-12-08 04:13:37,390 - DEBUG - Starting all layer 2 verification in agent 10.0.0.41
2025-12-08 04:13:37,401 - DEBUG - Route for VM MAC 02:f4:2d:69:5c:be is consistent in agent 10.0.0.41
2025-12-08 04:13:37,407 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is found in Agent of node 10.0.0.129
2025-12-08 04:13:37,544 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is consistent in agent 10.0.0.129
2025-12-08 04:13:37,545 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.129
2025-12-08 04:13:37,545 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 verification for VM ctest-vn1_vm1-39168219 in Agent 10.0.0.129 passed
2025-12-08 04:13:37,657 - DEBUG - Starting all layer 2 verification in agent 10.0.0.129
2025-12-08 04:13:37,669 - DEBUG - Route for VM MAC 02:f4:2d:69:5c:be is consistent in agent 10.0.0.129
2025-12-08 04:13:37,669 - INFO - VM ctest-vn1_vm1-39168219 verifications in Compute nodes passed
2025-12-08 04:13:37,917 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.41
2025-12-08 04:13:37,923 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.19, 24; Got 10.20.0.19,24
2025-12-08 04:13:37,923 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.41
2025-12-08 04:13:38,035 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.129
2025-12-08 04:13:38,039 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0
2025-12-08 04:13:38,039 - INFO - Validated routes of VM ctest-vn1_vm1-39168219 in all vrouters
2025-12-08 04:13:38,530 - DEBUG - Starting all layer2 verification in 10.0.0.22 Control Node
2025-12-08 04:13:38,658 - DEBUG - Layer2 route found for VM MAC 02:f4:2d:69:5c:be in Control-node 10.0.0.22
2025-12-08 04:13:38,658 - DEBUG - L2 Label for VM ctest-vn1_vm1-39168219 same between Control-node 10.0.0.22 and Agent, Expected: 29, Seen: 29
2025-12-08 04:13:38,658 - DEBUG - Starting all layer2 verification in 10.0.0.65 Control Node
2025-12-08 04:13:38,782 - DEBUG - Layer2 route found for VM MAC 02:f4:2d:69:5c:be in Control-node 10.0.0.65
2025-12-08 04:13:38,782 - DEBUG - L2 Label for VM ctest-vn1_vm1-39168219 same between Control-node 10.0.0.65 and Agent, Expected: 29, Seen: 29
2025-12-08 04:13:38,782 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node
2025-12-08 04:13:38,907 - DEBUG - Layer2 route found for VM MAC 02:f4:2d:69:5c:be in Control-node 10.0.0.23
2025-12-08 04:13:38,907 - DEBUG - L2 Label for VM ctest-vn1_vm1-39168219 same between Control-node 10.0.0.23 and Agent, Expected: 29, Seen: 29
2025-12-08 04:13:38,907 - INFO - Verification in Control-nodes for VM ctest-vn1_vm1-39168219 passed
2025-12-08 04:13:38,907 - DEBUG - Verifying the vm in opserver
2025-12-08 04:13:38,907 - DEBUG - Verifying in collector 10.0.0.65 ...
2025-12-08 04:13:38,907 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a?flat
2025-12-08 04:13:38,924 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-32026406:f42d695c-be91-4a87-9a71-288bd82ffd95?flat
2025-12-08 04:13:38,942 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-32026406:f42d695c-be91-4a87-9a71-288bd82ffd95?flat
2025-12-08 04:13:39,013 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a?flat
2025-12-08 04:13:39,082 - DEBUG - Verifying vm in vn uve
2025-12-08 04:13:39,082 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-32026406:f42d695c-be91-4a87-9a71-288bd82ffd95?flat
2025-12-08 04:13:39,152 - DEBUG - VM uve shows interface as {'admin_state': True, 'uuid': 'f42d695c-be91-4a87-9a71-288bd82ffd95', 'vhostuser_mode': 0, 'ip_address': '192.168.1.3', 'mac_address': '02:f4:2d:69:5c:be', 'vn_uuid': '3408134f-637d-48e3-9aee-4cb42988d140', 'vm_name': 'ctest-vn1_vm1-39168219', 'label': 24, 'active': True, 'ip4_active': True, 'fixed_ip4_list': ['192.168.1.3'], 'gateway': '192.168.1.1', 'is_health_check_active': True, 'l2_active': True, 'rx_vlan': 65535, 'tx_vlan': 65535, 'ip6_address': '::', 'port_mirror_enabled': False, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356', 'ip6_active': False, 'vm_uuid': 'a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a', '__T': 1765167078816424}
2025-12-08 04:13:39,152 - DEBUG - VM uve shows ip address as ['192.168.1.3', '::']
2025-12-08 04:13:39,152 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356
2025-12-08 04:13:39,152 - INFO - Verifying through opserver in 10.0.0.23
2025-12-08 04:13:39,152 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356?flat
2025-12-08 04:13:39,179 - DEBUG - expected vm list a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a
2025-12-08 04:13:39,179 - DEBUG - Extracted vm list ['a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a']
2025-12-08 04:13:39,179 - INFO - VM a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a is present in default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356
2025-12-08 04:13:39,179 - DEBUG - Verifying vm in vrouter uve
2025-12-08 04:13:39,179 - DEBUG - Getting info from collector 10.0.0.65..
2025-12-08 04:13:39,179 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a?flat
2025-12-08 04:13:39,247 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4854-2']
2025-12-08 04:13:39,247 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-2:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:13:39,285 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-2?flat
2025-12-08 04:13:39,315 - DEBUG - VM a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-2 uve
2025-12-08 04:13:39,316 - DEBUG - Validated that VM ctest-vn1_vm1-39168219 is in Vrouter an-jenkins-deploy-platform-ansible-os-4854-2 UVE
2025-12-08 04:13:39,316 - DEBUG - Verifying vm tap interface/vn in vrouter uve
2025-12-08 04:13:39,323 - DEBUG - Expected tap interface of VM uuid a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a is default-domain:ctest-TestBasicVMVN0-32026406:f42d695c-be91-4a87-9a71-288bd82ffd95
2025-12-08 04:13:39,323 - DEBUG - Expected VN of VM uuid a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a is default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356
2025-12-08 04:13:39,323 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-2:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:13:39,357 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-2?flat
2025-12-08 04:13:39,444 - DEBUG - VM a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-2 uve
2025-12-08 04:13:39,444 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-32026406:f42d695c-be91-4a87-9a71-288bd82ffd95 of vm a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-2 uve
2025-12-08 04:13:39,444 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 of vm a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-2 uve
2025-12-08 04:13:39,444 - DEBUG - Verifying the a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a virtual network link through opserver 10.0.0.23
2025-12-08 04:13:39,444 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines
2025-12-08 04:13:39,456 - DEBUG - VM link and name as {'name': 'a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a?flat'}
2025-12-08 04:13:39,456 - DEBUG - Verifying the a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a virtual network link through opserver 10.0.0.22
2025-12-08 04:13:39,456 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-12-08 04:13:39,520 - DEBUG - VM link and name as {'name': 'a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a?flat'}
2025-12-08 04:13:39,520 - DEBUG - Verifying the a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a virtual network link through opserver 10.0.0.65
2025-12-08 04:13:39,520 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machines
2025-12-08 04:13:39,534 - DEBUG - VM link and name as {'name': 'a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a', 'href': 'http://10.0.0.65:8081/analytics/uves/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a?flat'}
2025-12-08 04:13:39,534 - INFO - VM ctest-vn1_vm1-39168219 validations in Opserver passed
2025-12-08 04:13:39,612 - DEBUG - VM is in ACTIVE state now
2025-12-08 04:13:39,612 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:13:39,612 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/959d5c74-59a2-45f1-8a5c-1b768b83341b
2025-12-08 04:13:39,620 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/acc70189-933a-4697-9162-a8a2c71a9484
2025-12-08 04:13:39,629 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/959d5c74-59a2-45f1-8a5c-1b768b83341b
2025-12-08 04:13:39,637 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/acc70189-933a-4697-9162-a8a2c71a9484
2025-12-08 04:13:39,647 - DEBUG - Requesting: http://10.0.0.65:8082/instance-ip/6e0f59ff-20e5-4cbd-9e0c-ef6d43e32a62
2025-12-08 04:13:39,657 - DEBUG - Verifying in api server 10.0.0.65
2025-12-08 04:13:39,657 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/959d5c74-59a2-45f1-8a5c-1b768b83341b
2025-12-08 04:13:39,664 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/acc70189-933a-4697-9162-a8a2c71a9484
2025-12-08 04:13:39,673 - DEBUG - Requesting: http://10.0.0.65:8082/instance-ip/6e0f59ff-20e5-4cbd-9e0c-ef6d43e32a62
2025-12-08 04:13:39,681 - INFO - VM ctest-vn1_vm2-03488037 verfication in all API Servers passed
2025-12-08 04:13:39,697 - DEBUG - VM ctest-vn1_vm2-03488037 Tap interface: {'index': '3', 'name': 'tapacc70189-93', 'uuid': 'acc70189-933a-4697-9162-a8a2c71a9484', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356:ctest-vn1-49761356', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356', 'vm_uuid': '959d5c74-59a2-45f1-8a5c-1b768b83341b', 'vm_name': 'ctest-vn1_vm2-03488037', 'ip_addr': '192.168.1.4', 'mac_addr': '02:ac:c7:01:89:93', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '123', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-32026406:acc70189-933a-4697-9162-a8a2c71a9484', 'sg_uuid_list': ['a08ad7f7-4fb9-4642-b81a-e5edfb06e45f'], 'static_route_list': None, 'vm_project_uuid': '39b9363a-5400-4ea1-a180-7dd8e2fe11b9', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-12-08 04:13:39,697 - DEBUG - Agent 10.0.0.41 vrf name: default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356:ctest-vn1-49761356
2025-12-08 04:13:39,703 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356:ctest-vn1-49761356', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356', 'table_label': '-1', 'vxlan_id': '14', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.252:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]}
2025-12-08 04:13:39,782 - DEBUG - VM ctest-vn1_vm2-03488037 : Tap interface tapacc70189-93 is set to Active
2025-12-08 04:13:39,783 - DEBUG - Tap interface tapacc70189-93 detail : {'index': '3', 'name': 'tapacc70189-93', 'uuid': 'acc70189-933a-4697-9162-a8a2c71a9484', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356:ctest-vn1-49761356', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356', 'vm_uuid': '959d5c74-59a2-45f1-8a5c-1b768b83341b', 'vm_name': 'ctest-vn1_vm2-03488037', 'ip_addr': '192.168.1.4', 'mac_addr': '02:ac:c7:01:89:93', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '123', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-32026406:acc70189-933a-4697-9162-a8a2c71a9484', 'sg_uuid_list': ['a08ad7f7-4fb9-4642-b81a-e5edfb06e45f'], 'static_route_list': None, 'vm_project_uuid': '39b9363a-5400-4ea1-a180-7dd8e2fe11b9', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-12-08 04:13:39,894 - DEBUG - Starting Layer 2 verification in Agent
2025-12-08 04:13:39,905 - DEBUG - Layer 2 path is seen for VM MAC 02:ac:c7:01:89:93 in agent 10.0.0.41
2025-12-08 04:13:39,905 - DEBUG - Active layer 2 route in agent is present for VMI tapacc70189-93
2025-12-08 04:13:39,905 - DEBUG - L2 label(29) matches bw route table and interface table
2025-12-08 04:13:39,910 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is found in Agent of node 10.0.0.41
2025-12-08 04:13:40,042 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is consistent in agent 10.0.0.41
2025-12-08 04:13:40,042 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.41
2025-12-08 04:13:40,042 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 verification for VM ctest-vn1_vm2-03488037 in Agent 10.0.0.41 passed
2025-12-08 04:13:40,154 - DEBUG - Starting all layer 2 verification in agent 10.0.0.41
2025-12-08 04:13:40,165 - DEBUG - Route for VM MAC 02:ac:c7:01:89:93 is consistent in agent 10.0.0.41
2025-12-08 04:13:40,171 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is found in Agent of node 10.0.0.129
2025-12-08 04:13:40,305 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 is consistent in agent 10.0.0.129
2025-12-08 04:13:40,306 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.129
2025-12-08 04:13:40,306 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 verification for VM ctest-vn1_vm2-03488037 in Agent 10.0.0.129 passed
2025-12-08 04:13:40,417 - DEBUG - Starting all layer 2 verification in agent 10.0.0.129
2025-12-08 04:13:40,428 - DEBUG - Route for VM MAC 02:ac:c7:01:89:93 is consistent in agent 10.0.0.129
2025-12-08 04:13:40,428 - INFO - VM ctest-vn1_vm2-03488037 verifications in Compute nodes passed
2025-12-08 04:13:40,665 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.41
2025-12-08 04:13:40,670 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0
2025-12-08 04:13:40,782 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.129
2025-12-08 04:13:40,787 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.252, 24; Got 10.20.0.252,24
2025-12-08 04:13:40,787 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.129
2025-12-08 04:13:40,787 - INFO - Validated routes of VM ctest-vn1_vm2-03488037 in all vrouters
2025-12-08 04:13:41,315 - DEBUG - Starting all layer2 verification in 10.0.0.22 Control Node
2025-12-08 04:13:41,441 - DEBUG - Layer2 route found for VM MAC 02:ac:c7:01:89:93 in Control-node 10.0.0.22
2025-12-08 04:13:41,441 - DEBUG - L2 Label for VM ctest-vn1_vm2-03488037 same between Control-node 10.0.0.22 and Agent, Expected: 29, Seen: 29
2025-12-08 04:13:41,441 - DEBUG - Starting all layer2 verification in 10.0.0.65 Control Node
2025-12-08 04:13:41,571 - DEBUG - Layer2 route found for VM MAC 02:ac:c7:01:89:93 in Control-node 10.0.0.65
2025-12-08 04:13:41,571 - DEBUG - L2 Label for VM ctest-vn1_vm2-03488037 same between Control-node 10.0.0.65 and Agent, Expected: 29, Seen: 29
2025-12-08 04:13:41,571 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node
2025-12-08 04:13:41,704 - DEBUG - Layer2 route found for VM MAC 02:ac:c7:01:89:93 in Control-node 10.0.0.23
2025-12-08 04:13:41,705 - DEBUG - L2 Label for VM ctest-vn1_vm2-03488037 same between Control-node 10.0.0.23 and Agent, Expected: 29, Seen: 29
2025-12-08 04:13:41,705 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-03488037 passed
2025-12-08 04:13:41,705 - DEBUG - Verifying the vm in opserver
2025-12-08 04:13:41,705 - DEBUG - Verifying in collector 10.0.0.65 ...
2025-12-08 04:13:41,705 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machine/959d5c74-59a2-45f1-8a5c-1b768b83341b?flat
2025-12-08 04:13:41,809 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-32026406:acc70189-933a-4697-9162-a8a2c71a9484?flat
2025-12-08 04:13:41,831 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-32026406:acc70189-933a-4697-9162-a8a2c71a9484?flat
2025-12-08 04:13:41,855 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machine/959d5c74-59a2-45f1-8a5c-1b768b83341b?flat
2025-12-08 04:13:41,876 - DEBUG - Verifying vm in vn uve
2025-12-08 04:13:41,876 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-32026406:acc70189-933a-4697-9162-a8a2c71a9484?flat
2025-12-08 04:13:41,898 - DEBUG - VM uve shows interface as {'ip_address': '192.168.1.4', 'mac_address': '02:ac:c7:01:89:93', 'vm_name': 'ctest-vn1_vm2-03488037', 'admin_state': True, 'ip6_address': '::', 'rx_vlan': 65535, 'label': 24, 'gateway': '192.168.1.1', 'vm_uuid': '959d5c74-59a2-45f1-8a5c-1b768b83341b', 'is_health_check_active': True, 'active': True, 'uuid': 'acc70189-933a-4697-9162-a8a2c71a9484', '__T': 1765167100271520, 'ip6_active': False, 'ip4_active': True, 'l2_active': True, 'port_mirror_enabled': False, 'vhostuser_mode': 0, 'fixed_ip4_list': ['192.168.1.4'], 'tx_vlan': 65535, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356', 'vn_uuid': '3408134f-637d-48e3-9aee-4cb42988d140'}
2025-12-08 04:13:41,898 - DEBUG - VM uve shows ip address as ['192.168.1.4', '::']
2025-12-08 04:13:41,898 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356
2025-12-08 04:13:41,898 - INFO - Verifying through opserver in 10.0.0.23
2025-12-08 04:13:41,898 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356?flat
2025-12-08 04:13:41,927 - DEBUG - expected vm list 959d5c74-59a2-45f1-8a5c-1b768b83341b
2025-12-08 04:13:41,927 - DEBUG - Extracted vm list ['959d5c74-59a2-45f1-8a5c-1b768b83341b']
2025-12-08 04:13:41,927 - INFO - VM 959d5c74-59a2-45f1-8a5c-1b768b83341b is present in default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356
2025-12-08 04:13:41,927 - DEBUG - Verifying vm in vrouter uve
2025-12-08 04:13:41,927 - DEBUG - Getting info from collector 10.0.0.65..
2025-12-08 04:13:41,927 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machine/959d5c74-59a2-45f1-8a5c-1b768b83341b?flat
2025-12-08 04:13:41,945 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4854-1']
2025-12-08 04:13:41,945 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:13:41,995 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:13:42,029 - DEBUG - VM 959d5c74-59a2-45f1-8a5c-1b768b83341b is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:13:42,029 - DEBUG - Validated that VM ctest-vn1_vm2-03488037 is in Vrouter an-jenkins-deploy-platform-ansible-os-4854-1 UVE
2025-12-08 04:13:42,029 - DEBUG - Verifying vm tap interface/vn in vrouter uve
2025-12-08 04:13:42,038 - DEBUG - Expected tap interface of VM uuid 959d5c74-59a2-45f1-8a5c-1b768b83341b is default-domain:ctest-TestBasicVMVN0-32026406:acc70189-933a-4697-9162-a8a2c71a9484
2025-12-08 04:13:42,038 - DEBUG - Expected VN of VM uuid 959d5c74-59a2-45f1-8a5c-1b768b83341b is default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356
2025-12-08 04:13:42,038 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:13:42,088 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:13:42,127 - DEBUG - VM 959d5c74-59a2-45f1-8a5c-1b768b83341b is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:13:42,127 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-32026406:acc70189-933a-4697-9162-a8a2c71a9484 of vm 959d5c74-59a2-45f1-8a5c-1b768b83341b is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:13:42,127 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 of vm 959d5c74-59a2-45f1-8a5c-1b768b83341b is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:13:42,127 - DEBUG - Verifying the 959d5c74-59a2-45f1-8a5c-1b768b83341b virtual network link through opserver 10.0.0.23
2025-12-08 04:13:42,127 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines
2025-12-08 04:13:42,141 - DEBUG - VM link and name as {'name': '959d5c74-59a2-45f1-8a5c-1b768b83341b', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/959d5c74-59a2-45f1-8a5c-1b768b83341b?flat'}
2025-12-08 04:13:42,141 - DEBUG - Verifying the 959d5c74-59a2-45f1-8a5c-1b768b83341b virtual network link through opserver 10.0.0.22
2025-12-08 04:13:42,141 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-12-08 04:13:42,155 - DEBUG - VM link and name as {'name': '959d5c74-59a2-45f1-8a5c-1b768b83341b', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-machine/959d5c74-59a2-45f1-8a5c-1b768b83341b?flat'}
2025-12-08 04:13:42,155 - DEBUG - Verifying the 959d5c74-59a2-45f1-8a5c-1b768b83341b virtual network link through opserver 10.0.0.65
2025-12-08 04:13:42,155 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machines
2025-12-08 04:13:42,167 - DEBUG - VM link and name as {'name': '959d5c74-59a2-45f1-8a5c-1b768b83341b', 'href': 'http://10.0.0.65:8081/analytics/uves/virtual-machine/959d5c74-59a2-45f1-8a5c-1b768b83341b?flat'}
2025-12-08 04:13:42,167 - INFO - VM ctest-vn1_vm2-03488037 validations in Opserver passed
2025-12-08 04:13:42,167 - INFO - Checking the ping between the VM again
2025-12-08 04:13:42,167 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:13:42,168 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.129, gateway password: c0ntrail123
2025-12-08 04:13:42,168 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4
2025-12-08 04:13:44,670 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=3.89 ms
64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.09 ms
64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.23 ms
--- 192.168.1.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.091/2.075/3.899/1.291 ms
2025-12-08 04:13:44,670 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-39168219 passed
2025-12-08 04:13:44,670 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:13:44,670 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.41, gateway password: c0ntrail123
2025-12-08 04:13:44,670 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3
2025-12-08 04:13:47,126 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=3.74 ms
64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.50 ms
64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.48 ms
--- 192.168.1.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.488/2.244/3.745/1.061 ms
2025-12-08 04:13:47,126 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-03488037 passed
2025-12-08 04:13:47,126 - INFO - Deleting VM ctest-vn1_vm2-03488037
2025-12-08 04:13:47,200 - DEBUG - Verifying in api server 10.0.0.65
2025-12-08 04:13:47,200 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/959d5c74-59a2-45f1-8a5c-1b768b83341b
2025-12-08 04:13:47,208 - DEBUG - VM ID 959d5c74-59a2-45f1-8a5c-1b768b83341b of VM ctest-vn1_vm2-03488037 is still found in API Server
2025-12-08 04:13:49,209 - DEBUG - Verifying in api server 10.0.0.65
2025-12-08 04:13:49,209 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/959d5c74-59a2-45f1-8a5c-1b768b83341b
2025-12-08 04:13:49,217 - DEBUG - Response Code: 404
2025-12-08 04:13:49,217 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/959d5c74-59a2-45f1-8a5c-1b768b83341b
2025-12-08 04:13:49,224 - DEBUG - Response Code: 404
2025-12-08 04:13:49,224 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/959d5c74-59a2-45f1-8a5c-1b768b83341b
2025-12-08 04:13:49,231 - DEBUG - Response Code: 404
2025-12-08 04:13:49,231 - INFO - VM ctest-vn1_vm2-03488037 is fully removed in API-Server
2025-12-08 04:13:49,385 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found
2025-12-08 04:13:49,423 - DEBUG - Validated that vrouter 10.0.0.41 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356
2025-12-08 04:13:49,542 - DEBUG - VM route 192.168.1.4 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 of compute 10.0.0.129
2025-12-08 04:13:49,547 - DEBUG - Validated that vrouter 10.0.0.129 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356
2025-12-08 04:13:49,547 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356
2025-12-08 04:13:49,547 - INFO - VM ctest-vn1_vm2-03488037 is removed in Compute, and routes are removed in all compute nodes
2025-12-08 04:13:49,801 - INFO - Routes for VM ctest-vn1_vm2-03488037 is removed in all control-nodes
2025-12-08 04:13:49,801 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:13:49,854 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:13:49,886 - DEBUG - VM 959d5c74-59a2-45f1-8a5c-1b768b83341b is still present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:13:49,886 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:13:49,887 - DEBUG - interface for vm 959d5c74-59a2-45f1-8a5c-1b768b83341b deleted from vrouter uve ...
2025-12-08 04:13:49,887 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:13:52,888 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:13:52,949 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:13:52,981 - DEBUG - VM 959d5c74-59a2-45f1-8a5c-1b768b83341b is still present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:13:52,981 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:13:52,981 - DEBUG - interface for vm 959d5c74-59a2-45f1-8a5c-1b768b83341b deleted from vrouter uve ...
2025-12-08 04:13:52,981 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:13:55,982 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:13:56,031 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:13:56,063 - DEBUG - VM 959d5c74-59a2-45f1-8a5c-1b768b83341b is still present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:13:56,063 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:13:56,063 - DEBUG - interface for vm 959d5c74-59a2-45f1-8a5c-1b768b83341b deleted from vrouter uve ...
2025-12-08 04:13:56,063 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:13:59,064 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:13:59,127 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:13:59,161 - DEBUG - VM 959d5c74-59a2-45f1-8a5c-1b768b83341b is still present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:13:59,162 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:13:59,162 - DEBUG - interface for vm 959d5c74-59a2-45f1-8a5c-1b768b83341b deleted from vrouter uve ...
2025-12-08 04:13:59,162 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:14:02,163 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:14:02,212 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:14:02,250 - DEBUG - VM 959d5c74-59a2-45f1-8a5c-1b768b83341b is still present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:14:02,250 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:14:02,250 - DEBUG - interface for vm 959d5c74-59a2-45f1-8a5c-1b768b83341b deleted from vrouter uve ...
2025-12-08 04:14:02,250 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:14:05,251 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:14:05,328 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:14:05,374 - DEBUG - VM 959d5c74-59a2-45f1-8a5c-1b768b83341b is still present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:14:05,374 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:14:05,374 - DEBUG - interface for vm 959d5c74-59a2-45f1-8a5c-1b768b83341b deleted from vrouter uve ...
2025-12-08 04:14:05,374 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:14:08,374 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:14:08,432 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:14:08,465 - DEBUG - VM 959d5c74-59a2-45f1-8a5c-1b768b83341b is still present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:14:08,465 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:14:08,465 - DEBUG - interface for vm 959d5c74-59a2-45f1-8a5c-1b768b83341b deleted from vrouter uve ...
2025-12-08 04:14:08,465 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:14:11,466 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:14:11,565 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:14:11,613 - DEBUG - VM 959d5c74-59a2-45f1-8a5c-1b768b83341b is not present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:14:11,613 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:14:11,613 - DEBUG - interface for vm 959d5c74-59a2-45f1-8a5c-1b768b83341b deleted from vrouter uve ...
2025-12-08 04:14:11,613 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:14:11,613 - DEBUG - Validated that VM 959d5c74-59a2-45f1-8a5c-1b768b83341b is removed in Vrouter UVE
2025-12-08 04:14:11,613 - DEBUG - Verifying through opserver in 10.0.0.23
2025-12-08 04:14:11,613 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356?flat
2025-12-08 04:14:11,656 - DEBUG - 959d5c74-59a2-45f1-8a5c-1b768b83341b BM not in default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 uve
2025-12-08 04:14:11,656 - DEBUG - Validated that VM 959d5c74-59a2-45f1-8a5c-1b768b83341b is not present in the VN UVE
2025-12-08 04:14:11,656 - DEBUG - Verifying the 959d5c74-59a2-45f1-8a5c-1b768b83341b virtual network link through opserver 10.0.0.23
2025-12-08 04:14:11,656 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines
2025-12-08 04:14:11,677 - INFO - 959d5c74-59a2-45f1-8a5c-1b768b83341b vm uve deleted from opserver
2025-12-08 04:14:11,677 - DEBUG - Verifying the 959d5c74-59a2-45f1-8a5c-1b768b83341b virtual network link through opserver 10.0.0.22
2025-12-08 04:14:11,677 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-12-08 04:14:11,692 - INFO - 959d5c74-59a2-45f1-8a5c-1b768b83341b vm uve deleted from opserver
2025-12-08 04:14:11,693 - DEBUG - Verifying the 959d5c74-59a2-45f1-8a5c-1b768b83341b virtual network link through opserver 10.0.0.65
2025-12-08 04:14:11,693 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machines
2025-12-08 04:14:11,707 - INFO - 959d5c74-59a2-45f1-8a5c-1b768b83341b vm uve deleted from opserver
2025-12-08 04:14:11,707 - INFO - Validated that VM 959d5c74-59a2-45f1-8a5c-1b768b83341b is removed from Opserver
2025-12-08 04:14:11,707 - INFO - Deleting VM ctest-vn1_vm1-39168219
2025-12-08 04:14:11,798 - DEBUG - Verifying in api server 10.0.0.65
2025-12-08 04:14:11,799 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a
2025-12-08 04:14:11,807 - DEBUG - VM ID a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a of VM ctest-vn1_vm1-39168219 is still found in API Server
2025-12-08 04:14:13,808 - DEBUG - Verifying in api server 10.0.0.65
2025-12-08 04:14:13,808 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a
2025-12-08 04:14:13,815 - DEBUG - Response Code: 404
2025-12-08 04:14:13,815 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a
2025-12-08 04:14:13,820 - DEBUG - Response Code: 404
2025-12-08 04:14:13,820 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a
2025-12-08 04:14:13,826 - DEBUG - Response Code: 404
2025-12-08 04:14:13,826 - INFO - VM ctest-vn1_vm1-39168219 is fully removed in API-Server
2025-12-08 04:14:13,976 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found
2025-12-08 04:14:14,029 - DEBUG - Validated that vrouter 10.0.0.41 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356
2025-12-08 04:14:14,040 - DEBUG - Validated that vrouter 10.0.0.129 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356
2025-12-08 04:14:14,040 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356
2025-12-08 04:14:14,040 - INFO - VM ctest-vn1_vm1-39168219 is removed in Compute, and routes are removed in all compute nodes
2025-12-08 04:14:14,266 - INFO - Routes for VM ctest-vn1_vm1-39168219 is removed in all control-nodes
2025-12-08 04:14:14,266 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-2:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:14:14,303 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-2?flat
2025-12-08 04:14:14,334 - DEBUG - VM a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a is still present in vrouter an-jenkins-deploy-platform-ansible-os-4854-2 uve
2025-12-08 04:14:14,334 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:14:14,335 - DEBUG - interface for vm a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a deleted from vrouter uve ...
2025-12-08 04:14:14,335 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:14:17,336 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-2:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:14:17,376 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-2?flat
2025-12-08 04:14:17,408 - DEBUG - VM a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a is still present in vrouter an-jenkins-deploy-platform-ansible-os-4854-2 uve
2025-12-08 04:14:17,408 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:14:17,408 - DEBUG - interface for vm a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a deleted from vrouter uve ...
2025-12-08 04:14:17,408 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:14:20,409 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-2:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:14:20,450 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-2?flat
2025-12-08 04:14:20,497 - DEBUG - VM a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a is not present in vrouter an-jenkins-deploy-platform-ansible-os-4854-2 uve
2025-12-08 04:14:20,498 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:14:20,498 - DEBUG - interface for vm a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a deleted from vrouter uve ...
2025-12-08 04:14:20,498 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:14:20,498 - DEBUG - Validated that VM a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a is removed in Vrouter UVE
2025-12-08 04:14:20,498 - DEBUG - Verifying through opserver in 10.0.0.23
2025-12-08 04:14:20,498 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356?flat
2025-12-08 04:14:20,522 - DEBUG - a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a BM not in default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-49761356 uve
2025-12-08 04:14:20,522 - DEBUG - Validated that VM a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a is not present in the VN UVE
2025-12-08 04:14:20,522 - DEBUG - Verifying the a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a virtual network link through opserver 10.0.0.23
2025-12-08 04:14:20,522 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines
2025-12-08 04:14:20,535 - INFO - a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a vm uve deleted from opserver
2025-12-08 04:14:20,535 - DEBUG - Verifying the a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a virtual network link through opserver 10.0.0.22
2025-12-08 04:14:20,535 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-12-08 04:14:20,552 - INFO - a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a vm uve deleted from opserver
2025-12-08 04:14:20,552 - DEBUG - Verifying the a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a virtual network link through opserver 10.0.0.65
2025-12-08 04:14:20,552 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machines
2025-12-08 04:14:20,566 - INFO - a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a vm uve deleted from opserver
2025-12-08 04:14:20,566 - INFO - Validated that VM a1d77e8b-d4ef-4e0f-b4e9-4d2a2f33c56a is removed from Opserver
2025-12-08 04:14:20,566 - INFO - Deleting VN ctest-vn1-49761356
2025-12-08 04:14:20,789 - DEBUG - Response for deleting network ()
2025-12-08 04:14:20,790 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/e46942f0-8bc0-4b6a-951f-2a8248e1fbeb
2025-12-08 04:14:20,796 - DEBUG - Response Code: 404
2025-12-08 04:14:20,797 - DEBUG - Requesting: http://10.0.0.23:8082/domains
2025-12-08 04:14:20,803 - DEBUG - Requesting: http://10.0.0.23:8082/domain/c5befdc1-279e-4ac1-9620-9608231491f8
2025-12-08 04:14:20,869 - DEBUG - Requesting: http://10.0.0.23:8082/project/39b9363a-5400-4ea1-a180-7dd8e2fe11b9
2025-12-08 04:14:20,968 - INFO - Validated that VN ctest-vn1-49761356 is not found in API Server
2025-12-08 04:14:20,980 - DEBUG - VN ctest-vn1-49761356 is not present in Agent 10.0.0.41
2025-12-08 04:14:20,991 - DEBUG - VN ctest-vn1-49761356 is not present in Agent 10.0.0.129
2025-12-08 04:14:20,991 - INFO - Validated that VN ctest-vn1-49761356 is not in any agent
2025-12-08 04:14:21,028 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-49761356 info
2025-12-08 04:14:22,296 - DEBUG - Skipping xmpp flap check
2025-12-08 04:14:22,296 - INFO - END TEST : test_control_node_switchover : PASSED[0:03:20]
2025-12-08 04:14:22,297 - INFO - --------------------------------------------------------------------------------
2025-12-08 04:14:22,302 - INFO - ================================================================================
2025-12-08 04:14:22,303 - INFO - STARTING TEST : test_process_restart_in_policy_between_vns
2025-12-08 04:14:22,304 - INFO - TEST DESCRIPTION : Test to validate that with policy having rule to check icmp fwding between VMs on different VNs , ping between VMs should pass
with process restarts
1. Pick 2 VN's from resource pool which has one VM each
2. Create policy with icmp allow rule between those VN's and bind it networks
3. Ping from one VM to another VM
4. Restart process 'vrouter' and 'control' on setup
5. Ping again between VM's after process restart
Pass criteria: Step 2,3,4 and 5 should pass
2025-12-08 04:14:23,587 - DEBUG - Skipping xmpp flap check
2025-12-08 04:14:23,587 - INFO - Initial checks done. Running the testcase now
2025-12-08 04:14:23,587 - INFO -
2025-12-08 04:14:23,593 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None]
2025-12-08 04:14:23,901 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None]
2025-12-08 04:14:24,445 - INFO - Created VN ctest-vn1-11209066, UUID :75a3447b-83a9-421f-b144-cf08e0cc11a5
2025-12-08 04:14:24,758 - DEBUG - Requesting: http://10.0.0.23:8082/domains
2025-12-08 04:14:24,765 - DEBUG - Requesting: http://10.0.0.23:8082/domain/c5befdc1-279e-4ac1-9620-9608231491f8
2025-12-08 04:14:24,776 - DEBUG - Requesting: http://10.0.0.23:8082/project/39b9363a-5400-4ea1-a180-7dd8e2fe11b9
2025-12-08 04:14:24,869 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/75a3447b-83a9-421f-b144-cf08e0cc11a5
2025-12-08 04:14:24,889 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/75a3447b-83a9-421f-b144-cf08e0cc11a5
2025-12-08 04:14:24,899 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/904d4940-9135-42bf-812f-06d264925cbe
2025-12-08 04:14:24,908 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/904d4940-9135-42bf-812f-06d264925cbe
2025-12-08 04:14:24,916 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/3ddfeaf6-800b-4608-a01a-f15599d0b6a6
2025-12-08 04:14:24,922 - DEBUG - Route Targets: ['target:64512:8000004']
2025-12-08 04:14:24,922 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/75a3447b-83a9-421f-b144-cf08e0cc11a5
2025-12-08 04:14:24,931 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/904d4940-9135-42bf-812f-06d264925cbe
2025-12-08 04:14:24,946 - INFO - Verified VN network id 14 for VN 75a3447b-83a9-421f-b144-cf08e0cc11a5
2025-12-08 04:14:24,946 - INFO - Verifications in API Server for VN ctest-vn1-11209066 passed
2025-12-08 04:14:24,946 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/75a3447b-83a9-421f-b144-cf08e0cc11a5
2025-12-08 04:14:24,956 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/904d4940-9135-42bf-812f-06d264925cbe
2025-12-08 04:14:24,964 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/904d4940-9135-42bf-812f-06d264925cbe
2025-12-08 04:14:24,971 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/3ddfeaf6-800b-4608-a01a-f15599d0b6a6
2025-12-08 04:14:24,982 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-32026406', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '8476694220921979423', 'uuid-lslong': '12773562080167727525'}, 'enable': 'true', 'created': '2025-12-08T04:14:24', 'last-modified': '2025-12-08T04:14:24', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.393762'}
2025-12-08 04:14:24,987 - DEBUG - Route Targets: ['target:64512:8000004']
2025-12-08 04:14:24,993 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-32026406', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '8476694220921979423', 'uuid-lslong': '12773562080167727525'}, 'enable': 'true', 'created': '2025-12-08T04:14:24', 'last-modified': '2025-12-08T04:14:24', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.413566'}
2025-12-08 04:14:25,000 - DEBUG - Route Targets: ['target:64512:8000004']
2025-12-08 04:14:25,006 - DEBUG - Control-node 10.0.0.65 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-32026406', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '8476694220921979423', 'uuid-lslong': '12773562080167727525'}, 'enable': 'true', 'created': '2025-12-08T04:14:24', 'last-modified': '2025-12-08T04:14:24', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.417479'}
2025-12-08 04:14:25,011 - DEBUG - Route Targets: ['target:64512:8000004']
2025-12-08 04:14:25,011 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-11209066 passed
2025-12-08 04:14:25,011 - DEBUG - ====Verifying policy data for ctest-vn1-11209066 in API_Server ======
2025-12-08 04:14:25,011 - DEBUG - Requesting: http://10.0.0.23:8082/domains
2025-12-08 04:14:25,017 - DEBUG - Requesting: http://10.0.0.23:8082/domain/c5befdc1-279e-4ac1-9620-9608231491f8
2025-12-08 04:14:25,030 - DEBUG - Requesting: http://10.0.0.23:8082/project/39b9363a-5400-4ea1-a180-7dd8e2fe11b9
2025-12-08 04:14:25,124 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/75a3447b-83a9-421f-b144-cf08e0cc11a5
2025-12-08 04:14:25,135 - DEBUG - =>VN ctest-vn1-11209066 has no policy to be verified
2025-12-08 04:14:25,135 - DEBUG - Verifying the vn in opserver
2025-12-08 04:14:25,135 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 virtual network link through opserver 10.0.0.23
2025-12-08 04:14:25,135 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks
2025-12-08 04:14:25,147 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066?flat'}
2025-12-08 04:14:25,147 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 virtual network link through opserver 10.0.0.22
2025-12-08 04:14:25,147 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks
2025-12-08 04:14:25,161 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066?flat'}
2025-12-08 04:14:25,161 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 virtual network link through opserver 10.0.0.65
2025-12-08 04:14:25,161 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-networks
2025-12-08 04:14:25,174 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'href': 'http://10.0.0.65:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066?flat'}
2025-12-08 04:14:25,174 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is found in opserver
2025-12-08 04:14:25,196 - DEBUG - Do not have enough data to verify VN in agent
2025-12-08 04:14:25,209 - DEBUG - VRF ids for VN ctest-vn1-11209066: {}
2025-12-08 04:14:25,877 - INFO - Created VN ctest-vn2-01263834, UUID :209f34d1-dfa3-4731-a388-f9ca87d11aad
2025-12-08 04:14:26,212 - DEBUG - Requesting: http://10.0.0.23:8082/domains
2025-12-08 04:14:26,218 - DEBUG - Requesting: http://10.0.0.23:8082/domain/c5befdc1-279e-4ac1-9620-9608231491f8
2025-12-08 04:14:26,229 - DEBUG - Requesting: http://10.0.0.23:8082/project/39b9363a-5400-4ea1-a180-7dd8e2fe11b9
2025-12-08 04:14:26,323 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/209f34d1-dfa3-4731-a388-f9ca87d11aad
2025-12-08 04:14:26,335 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/209f34d1-dfa3-4731-a388-f9ca87d11aad
2025-12-08 04:14:26,345 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/8454ee1b-3fd7-41ef-8a6d-c483d648abff
2025-12-08 04:14:26,355 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/8454ee1b-3fd7-41ef-8a6d-c483d648abff
2025-12-08 04:14:26,365 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/8d5724c4-cee4-4229-b241-0d78569c7aad
2025-12-08 04:14:26,374 - DEBUG - Route Targets: ['target:64512:8000005']
2025-12-08 04:14:26,374 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/209f34d1-dfa3-4731-a388-f9ca87d11aad
2025-12-08 04:14:26,385 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/8454ee1b-3fd7-41ef-8a6d-c483d648abff
2025-12-08 04:14:26,403 - INFO - Verified VN network id 15 for VN 209f34d1-dfa3-4731-a388-f9ca87d11aad
2025-12-08 04:14:26,403 - INFO - Verifications in API Server for VN ctest-vn2-01263834 passed
2025-12-08 04:14:26,403 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/209f34d1-dfa3-4731-a388-f9ca87d11aad
2025-12-08 04:14:26,414 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/8454ee1b-3fd7-41ef-8a6d-c483d648abff
2025-12-08 04:14:26,422 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/8454ee1b-3fd7-41ef-8a6d-c483d648abff
2025-12-08 04:14:26,433 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/8d5724c4-cee4-4229-b241-0d78569c7aad
2025-12-08 04:14:26,448 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-32026406', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2350655606515517233', 'uuid-lslong': '11783943073272240813'}, 'enable': 'true', 'created': '2025-12-08T04:14:25', 'last-modified': '2025-12-08T04:14:26', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.420064'}
2025-12-08 04:14:26,454 - DEBUG - Route Targets: ['target:64512:8000005']
2025-12-08 04:14:26,460 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-32026406', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2350655606515517233', 'uuid-lslong': '11783943073272240813'}, 'enable': 'true', 'created': '2025-12-08T04:14:25', 'last-modified': '2025-12-08T04:14:26', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.430906'}
2025-12-08 04:14:26,466 - DEBUG - Route Targets: ['target:64512:8000005']
2025-12-08 04:14:26,472 - DEBUG - Control-node 10.0.0.65 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-32026406', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '2350655606515517233', 'uuid-lslong': '11783943073272240813'}, 'enable': 'true', 'created': '2025-12-08T04:14:25', 'last-modified': '2025-12-08T04:14:26', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.441166'}
2025-12-08 04:14:26,477 - DEBUG - Route Targets: ['target:64512:8000005']
2025-12-08 04:14:26,478 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-01263834 passed
2025-12-08 04:14:26,478 - DEBUG - ====Verifying policy data for ctest-vn2-01263834 in API_Server ======
2025-12-08 04:14:26,478 - DEBUG - Requesting: http://10.0.0.23:8082/domains
2025-12-08 04:14:26,486 - DEBUG - Requesting: http://10.0.0.23:8082/domain/c5befdc1-279e-4ac1-9620-9608231491f8
2025-12-08 04:14:26,504 - DEBUG - Requesting: http://10.0.0.23:8082/project/39b9363a-5400-4ea1-a180-7dd8e2fe11b9
2025-12-08 04:14:26,593 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/209f34d1-dfa3-4731-a388-f9ca87d11aad
2025-12-08 04:14:26,610 - DEBUG - =>VN ctest-vn2-01263834 has no policy to be verified
2025-12-08 04:14:26,610 - DEBUG - Verifying the vn in opserver
2025-12-08 04:14:26,610 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 virtual network link through opserver 10.0.0.23
2025-12-08 04:14:26,610 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks
2025-12-08 04:14:26,622 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834?flat'}
2025-12-08 04:14:26,622 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 virtual network link through opserver 10.0.0.22
2025-12-08 04:14:26,622 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks
2025-12-08 04:14:26,636 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834?flat'}
2025-12-08 04:14:26,636 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 virtual network link through opserver 10.0.0.65
2025-12-08 04:14:26,636 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-networks
2025-12-08 04:14:26,652 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834', 'href': 'http://10.0.0.65:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834?flat'}
2025-12-08 04:14:26,652 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 is found in opserver
2025-12-08 04:14:26,673 - DEBUG - Do not have enough data to verify VN in agent
2025-12-08 04:14:26,684 - DEBUG - VRF ids for VN ctest-vn2-01263834: {}
2025-12-08 04:14:28,318 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4854-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4854-2)
2025-12-08 04:14:29,871 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4854-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4854-1)
2025-12-08 04:14:29,871 - INFO - Waiting for VM ctest-vn1_vm1-42592862 to be up..
2025-12-08 04:14:29,956 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2025-12-08 04:14:35,040 - DEBUG - VM is in ACTIVE state now
2025-12-08 04:14:35,040 - INFO - VM name : ctest-vn1_vm1-42592862
2025-12-08 04:14:35,123 - DEBUG - VM ctest-vn1_vm1-42592862 ID is 45297b8d-047f-460c-abcc-9e49a11b86fa
2025-12-08 04:14:35,123 - DEBUG - VM ctest-vn1_vm1-42592862 launched on Node an-jenkins-deploy-platform-ansible-os-4854-2
2025-12-08 04:14:35,199 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/45297b8d-047f-460c-abcc-9e49a11b86fa
2025-12-08 04:14:35,210 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/08e69f14-91f8-4e4a-b704-fc5edd56e50b
2025-12-08 04:14:38,396 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1028ms')
2025-12-08 04:14:38,396 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42592862 failed!
2025-12-08 04:14:38,410 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:14:42,489 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms')
2025-12-08 04:14:42,489 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42592862 failed!
2025-12-08 04:14:42,503 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:14:46,590 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms')
2025-12-08 04:14:46,590 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42592862 failed!
2025-12-08 04:14:46,605 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:14:50,690 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms')
2025-12-08 04:14:50,690 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42592862 failed!
2025-12-08 04:14:50,703 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:14:54,777 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms')
2025-12-08 04:14:54,777 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42592862 failed!
2025-12-08 04:14:54,795 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:14:58,882 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms')
2025-12-08 04:14:58,882 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42592862 failed!
2025-12-08 04:14:58,896 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:15:02,971 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms')
2025-12-08 04:15:02,971 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42592862 failed!
2025-12-08 04:15:02,985 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:15:07,071 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms')
2025-12-08 04:15:07,071 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42592862 failed!
2025-12-08 04:15:07,086 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:15:11,170 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms')
2025-12-08 04:15:11,170 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42592862 failed!
2025-12-08 04:15:11,186 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:15:13,250 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=11.4 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=2.83 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 2.831/7.128/11.425/4.297 ms')
2025-12-08 04:15:13,250 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-42592862 passed
2025-12-08 04:15:13,318 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:15:13,318 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-42592862, IP 192.168.1.252, Port 22
2025-12-08 04:15:13,386 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-12-08 04:15:13,474 - DEBUG - VM ctest-vn1_vm1-42592862 is NOT ready for SSH connections, VM status: ACTIVE
2025-12-08 04:15:18,475 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:15:18,475 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-42592862, IP 192.168.1.252, Port 22
2025-12-08 04:15:18,542 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-12-08 04:15:18,623 - DEBUG - VM ctest-vn1_vm1-42592862 is NOT ready for SSH connections, VM status: ACTIVE
2025-12-08 04:15:23,625 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:15:23,625 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-42592862, IP 192.168.1.252, Port 22
2025-12-08 04:15:23,694 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-12-08 04:15:23,796 - DEBUG - VM ctest-vn1_vm1-42592862 is NOT ready for SSH connections, VM status: ACTIVE
2025-12-08 04:15:28,797 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:15:28,797 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-42592862, IP 192.168.1.252, Port 22
2025-12-08 04:15:28,869 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-12-08 04:15:28,952 - DEBUG - VM ctest-vn1_vm1-42592862 is NOT ready for SSH connections, VM status: ACTIVE
2025-12-08 04:15:33,953 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:15:33,953 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-42592862, IP 192.168.1.252, Port 22
2025-12-08 04:15:34,121 - DEBUG - VM ctest-vn1_vm1-42592862 is ready for SSH connections
2025-12-08 04:15:34,121 - INFO - Waiting for VM ctest-vn2_vm1-91232456 to be up..
2025-12-08 04:15:34,202 - DEBUG - VM is in ACTIVE state now
2025-12-08 04:15:34,202 - INFO - VM name : ctest-vn2_vm1-91232456
2025-12-08 04:15:34,284 - DEBUG - VM ctest-vn2_vm1-91232456 ID is 245cd8ec-6d0a-4681-8d0d-96fa294496a7
2025-12-08 04:15:34,284 - DEBUG - VM ctest-vn2_vm1-91232456 launched on Node an-jenkins-deploy-platform-ansible-os-4854-1
2025-12-08 04:15:34,360 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/245cd8ec-6d0a-4681-8d0d-96fa294496a7
2025-12-08 04:15:34,369 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/37e4768a-4089-4b09-a908-a40b64820359
2025-12-08 04:15:35,527 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.61 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.934 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.934/3.273/5.612/2.339 ms')
2025-12-08 04:15:35,527 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm1-91232456 passed
2025-12-08 04:15:35,597 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:15:35,598 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-91232456, IP 192.168.2.252, Port 22
2025-12-08 04:15:35,769 - DEBUG - VM ctest-vn2_vm1-91232456 is ready for SSH connections
2025-12-08 04:15:35,769 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:15:35,769 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.252, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.129, gateway password: c0ntrail123
2025-12-08 04:15:35,769 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.252
2025-12-08 04:15:58,972 - DEBUG - PING 192.168.2.252 (192.168.2.252) 56(84) bytes of data.
64 bytes from 192.168.2.252: icmp_req=1 ttl=63 time=4.99 ms
64 bytes from 192.168.2.252: icmp_req=2 ttl=63 time=1.30 ms
64 bytes from 192.168.2.252: icmp_req=3 ttl=63 time=1.15 ms
--- 192.168.2.252 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.156/2.485/4.999/1.778 ms
2025-12-08 04:15:58,972 - INFO - Ping to IP 192.168.2.252 from VM ctest-vn1_vm1-42592862 passed
2025-12-08 04:16:02,413 - INFO - 10.0.0.41
2025-12-08 04:16:02,417 - INFO - agent:initializing
2025-12-08 04:16:02,418 - DEBUG - defaultdict(, {'10.0.0.41': {'agent': {'status': 'initializing', 'description': None}}})
2025-12-08 04:16:02,418 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-12-08 04:16:07,529 - INFO - 10.0.0.41
2025-12-08 04:16:07,534 - INFO - agent:active
2025-12-08 04:16:07,535 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.41
2025-12-08 04:16:10,927 - INFO - 10.0.0.129
2025-12-08 04:16:10,931 - INFO - agent:initializing
2025-12-08 04:16:10,931 - DEBUG - defaultdict(, {'10.0.0.129': {'agent': {'status': 'initializing', 'description': None}}})
2025-12-08 04:16:10,931 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-12-08 04:16:16,019 - INFO - 10.0.0.129
2025-12-08 04:16:16,025 - INFO - agent:active
2025-12-08 04:16:16,026 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.129
2025-12-08 04:16:16,625 - INFO - 10.0.0.23
2025-12-08 04:16:16,628 - INFO - control:initializing
2025-12-08 04:16:16,628 - DEBUG - defaultdict(, {'10.0.0.23': {'control': {'status': 'initializing', 'description': None}}})
2025-12-08 04:16:16,628 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-12-08 04:16:21,741 - INFO - 10.0.0.23
2025-12-08 04:16:21,748 - INFO - control:active
2025-12-08 04:16:21,748 - INFO - Contrail services ['control'] are up on nodes 10.0.0.23
2025-12-08 04:16:22,139 - INFO - 10.0.0.22
2025-12-08 04:16:22,143 - INFO - control:initializing
2025-12-08 04:16:22,143 - DEBUG - defaultdict(, {'10.0.0.22': {'control': {'status': 'initializing', 'description': None}}})
2025-12-08 04:16:22,143 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-12-08 04:16:27,244 - INFO - 10.0.0.22
2025-12-08 04:16:27,251 - INFO - control:active
2025-12-08 04:16:27,251 - INFO - Contrail services ['control'] are up on nodes 10.0.0.22
2025-12-08 04:16:27,689 - INFO - 10.0.0.65
2025-12-08 04:16:27,694 - INFO - control:initializing
2025-12-08 04:16:27,694 - DEBUG - defaultdict(, {'10.0.0.65': {'control': {'status': 'initializing', 'description': None}}})
2025-12-08 04:16:27,694 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-12-08 04:16:32,797 - INFO - 10.0.0.65
2025-12-08 04:16:32,804 - INFO - control:active
2025-12-08 04:16:32,804 - INFO - Contrail services ['control'] are up on nodes 10.0.0.65
2025-12-08 04:16:33,533 - INFO - 10.0.0.23
2025-12-08 04:16:33,536 - INFO - api-server:initializing
2025-12-08 04:16:33,536 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': None}}})
2025-12-08 04:16:33,536 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-12-08 04:16:38,665 - INFO - 10.0.0.23
2025-12-08 04:16:38,668 - INFO - api-server:initializing
2025-12-08 04:16:38,669 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': None}}})
2025-12-08 04:16:38,669 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1
2025-12-08 04:16:43,798 - INFO - 10.0.0.23
2025-12-08 04:16:43,804 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down)
2025-12-08 04:16:43,804 - DEBUG - defaultdict(, {'10.0.0.23': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}})
2025-12-08 04:16:43,804 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2
2025-12-08 04:16:48,919 - INFO - 10.0.0.23
2025-12-08 04:16:48,925 - INFO - api-server:active
2025-12-08 04:16:48,925 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.23
2025-12-08 04:16:49,601 - INFO - 10.0.0.22
2025-12-08 04:16:49,605 - INFO - api-server:initializing
2025-12-08 04:16:49,605 - DEBUG - defaultdict(, {'10.0.0.22': {'api-server': {'status': 'initializing', 'description': None}}})
2025-12-08 04:16:49,605 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-12-08 04:16:54,739 - INFO - 10.0.0.22
2025-12-08 04:16:54,743 - INFO - api-server:initializing
2025-12-08 04:16:54,744 - DEBUG - defaultdict(, {'10.0.0.22': {'api-server': {'status': 'initializing', 'description': None}}})
2025-12-08 04:16:54,744 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1
2025-12-08 04:16:59,845 - INFO - 10.0.0.22
2025-12-08 04:16:59,852 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down)
2025-12-08 04:16:59,852 - DEBUG - defaultdict(, {'10.0.0.22': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}})
2025-12-08 04:16:59,852 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2
2025-12-08 04:17:04,953 - INFO - 10.0.0.22
2025-12-08 04:17:04,958 - INFO - api-server:active
2025-12-08 04:17:04,958 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.22
2025-12-08 04:17:05,636 - INFO - 10.0.0.65
2025-12-08 04:17:05,640 - INFO - api-server:initializing
2025-12-08 04:17:05,640 - DEBUG - defaultdict(, {'10.0.0.65': {'api-server': {'status': 'initializing', 'description': None}}})
2025-12-08 04:17:05,640 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-12-08 04:17:10,754 - INFO - 10.0.0.65
2025-12-08 04:17:10,759 - INFO - api-server:initializing
2025-12-08 04:17:10,759 - DEBUG - defaultdict(, {'10.0.0.65': {'api-server': {'status': 'initializing', 'description': None}}})
2025-12-08 04:17:10,759 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1
2025-12-08 04:17:15,853 - INFO - 10.0.0.65
2025-12-08 04:17:15,859 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down)
2025-12-08 04:17:15,859 - DEBUG - defaultdict(, {'10.0.0.65': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}})
2025-12-08 04:17:15,859 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2
2025-12-08 04:17:20,966 - INFO - 10.0.0.65
2025-12-08 04:17:20,971 - INFO - api-server:active
2025-12-08 04:17:20,971 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.65
2025-12-08 04:17:21,093 - INFO - 10.0.0.23
2025-12-08 04:17:21,093 - INFO - config
2025-12-08 04:17:21,099 - INFO - config-nodemgr:active
2025-12-08 04:17:21,103 - INFO - api-server:active
2025-12-08 04:17:21,107 - INFO - schema:backup
2025-12-08 04:17:21,109 - INFO - svc-monitor:backup
2025-12-08 04:17:21,112 - INFO - device-manager:backup
2025-12-08 04:17:21,112 - INFO - config-database
2025-12-08 04:17:21,112 - INFO - config-cassandra:active
2025-12-08 04:17:21,112 - INFO - config-zookeeper:active
2025-12-08 04:17:21,112 - INFO - config-rabbitmq:active
2025-12-08 04:17:21,112 - INFO - control
2025-12-08 04:17:21,117 - INFO - control-nodemgr:active
2025-12-08 04:17:21,122 - INFO - control:active
2025-12-08 04:17:21,122 - INFO - named:active
2025-12-08 04:17:21,128 - INFO - dns:active
2025-12-08 04:17:21,128 - INFO - analytics
2025-12-08 04:17:21,132 - INFO - analytics-nodemgr:active
2025-12-08 04:17:21,136 - INFO - analytics-api:active
2025-12-08 04:17:21,140 - INFO - collector:active
2025-12-08 04:17:21,140 - INFO - analytics-database
2025-12-08 04:17:21,141 - INFO - analytics-cassandra:active
2025-12-08 04:17:21,158 - INFO - analyticsdb-nodemgr:active
2025-12-08 04:17:21,163 - INFO - query-engine:active
2025-12-08 04:17:21,164 - INFO - webui
2025-12-08 04:17:21,164 - INFO - webui:active
2025-12-08 04:17:21,164 - INFO - webui-middleware:active
2025-12-08 04:17:21,164 - INFO - redis:active
2025-12-08 04:17:21,164 - INFO - analytics_snmp
2025-12-08 04:17:21,169 - INFO - snmp-collector:active
2025-12-08 04:17:21,169 - INFO - snmp-topology:active
2025-12-08 04:17:21,169 - INFO - analytics_alarm
2025-12-08 04:17:21,169 - INFO - alarmgen:active
2025-12-08 04:17:21,268 - INFO - 10.0.0.22
2025-12-08 04:17:21,269 - INFO - config
2025-12-08 04:17:21,274 - INFO - config-nodemgr:active
2025-12-08 04:17:21,279 - INFO - api-server:active
2025-12-08 04:17:21,285 - INFO - schema:active
2025-12-08 04:17:21,290 - INFO - svc-monitor:active
2025-12-08 04:17:21,296 - INFO - device-manager:active
2025-12-08 04:17:21,296 - INFO - config-database
2025-12-08 04:17:21,296 - INFO - config-cassandra:active
2025-12-08 04:17:21,296 - INFO - config-zookeeper:active
2025-12-08 04:17:21,297 - INFO - config-rabbitmq:active
2025-12-08 04:17:21,297 - INFO - control
2025-12-08 04:17:21,303 - INFO - control-nodemgr:active
2025-12-08 04:17:21,310 - INFO - control:active
2025-12-08 04:17:21,310 - INFO - named:active
2025-12-08 04:17:21,315 - INFO - dns:active
2025-12-08 04:17:21,315 - INFO - analytics
2025-12-08 04:17:21,320 - INFO - analytics-nodemgr:active
2025-12-08 04:17:21,325 - INFO - analytics-api:active
2025-12-08 04:17:21,330 - INFO - collector:active
2025-12-08 04:17:21,331 - INFO - analytics-database
2025-12-08 04:17:21,331 - INFO - analytics-cassandra:active
2025-12-08 04:17:21,337 - INFO - analyticsdb-nodemgr:active
2025-12-08 04:17:21,344 - INFO - query-engine:active
2025-12-08 04:17:21,344 - INFO - webui
2025-12-08 04:17:21,344 - INFO - webui:active
2025-12-08 04:17:21,344 - INFO - webui-middleware:active
2025-12-08 04:17:21,344 - INFO - redis:active
2025-12-08 04:17:21,344 - INFO - analytics_snmp
2025-12-08 04:17:21,348 - INFO - snmp-collector:active
2025-12-08 04:17:21,348 - INFO - snmp-topology:active
2025-12-08 04:17:21,348 - INFO - analytics_alarm
2025-12-08 04:17:21,348 - INFO - alarmgen:active
2025-12-08 04:17:21,448 - INFO - 10.0.0.65
2025-12-08 04:17:21,448 - INFO - config
2025-12-08 04:17:21,454 - INFO - config-nodemgr:active
2025-12-08 04:17:21,458 - INFO - api-server:active
2025-12-08 04:17:21,461 - INFO - schema:backup
2025-12-08 04:17:21,465 - INFO - svc-monitor:backup
2025-12-08 04:17:21,469 - INFO - device-manager:backup
2025-12-08 04:17:21,469 - INFO - config-database
2025-12-08 04:17:21,469 - INFO - config-cassandra:active
2025-12-08 04:17:21,469 - INFO - config-zookeeper:active
2025-12-08 04:17:21,469 - INFO - config-rabbitmq:active
2025-12-08 04:17:21,469 - INFO - control
2025-12-08 04:17:21,474 - INFO - control-nodemgr:active
2025-12-08 04:17:21,480 - INFO - control:active
2025-12-08 04:17:21,480 - INFO - named:active
2025-12-08 04:17:21,485 - INFO - dns:active
2025-12-08 04:17:21,485 - INFO - analytics
2025-12-08 04:17:21,490 - INFO - analytics-nodemgr:active
2025-12-08 04:17:21,494 - INFO - analytics-api:active
2025-12-08 04:17:21,499 - INFO - collector:active
2025-12-08 04:17:21,499 - INFO - analytics-database
2025-12-08 04:17:21,499 - INFO - analytics-cassandra:active
2025-12-08 04:17:21,504 - INFO - analyticsdb-nodemgr:active
2025-12-08 04:17:21,509 - INFO - query-engine:active
2025-12-08 04:17:21,509 - INFO - webui
2025-12-08 04:17:21,509 - INFO - webui:active
2025-12-08 04:17:21,510 - INFO - webui-middleware:active
2025-12-08 04:17:21,510 - INFO - redis:active
2025-12-08 04:17:21,510 - INFO - analytics_snmp
2025-12-08 04:17:21,515 - INFO - snmp-collector:active
2025-12-08 04:17:21,515 - INFO - snmp-topology:active
2025-12-08 04:17:21,515 - INFO - analytics_alarm
2025-12-08 04:17:21,515 - INFO - alarmgen:active
2025-12-08 04:17:21,613 - INFO - 10.0.0.41
2025-12-08 04:17:21,613 - INFO - vrouter
2025-12-08 04:17:21,618 - INFO - vrouter-nodemgr:active
2025-12-08 04:17:21,623 - INFO - agent:active
2025-12-08 04:17:21,710 - INFO - 10.0.0.129
2025-12-08 04:17:21,711 - INFO - vrouter
2025-12-08 04:17:21,716 - INFO - vrouter-nodemgr:active
2025-12-08 04:17:21,721 - INFO - agent:active
2025-12-08 04:17:21,721 - INFO - All the contrail services are up on all nodes
2025-12-08 04:17:21,721 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:17:22,160 - INFO - an-jenkins-deploy-platform-ansible-os-4854-1:contrail-vrouter-agent module connection to ['10.20.0.18:5269'] servers UP
2025-12-08 04:17:22,160 - INFO - an-jenkins-deploy-platform-ansible-os-4854-1 connected to xmpp 10.20.0.18
2025-12-08 04:17:22,160 - WARNING - an-jenkins-deploy-platform-ansible-os-4854-1:contrail-vrouter-agent module connection to ['10.20.0.14:5269'] servers NOT UP
2025-12-08 04:17:22,160 - INFO - an-jenkins-deploy-platform-ansible-os-4854-1:contrail-vrouter-agent module connection to ['10.20.0.145:5269'] servers UP
2025-12-08 04:17:22,160 - INFO - an-jenkins-deploy-platform-ansible-os-4854-1 connected to xmpp 10.20.0.145
2025-12-08 04:17:22,160 - INFO - an-jenkins-deploy-platform-ansible-os-4854-1:contrail-vrouter-agent module connection to ['10.20.0.18:53'] servers UP
2025-12-08 04:17:22,161 - INFO - an-jenkins-deploy-platform-ansible-os-4854-1 connected to dns 10.20.0.18
2025-12-08 04:17:22,161 - INFO - an-jenkins-deploy-platform-ansible-os-4854-1:contrail-vrouter-agent module connection to ['10.20.0.14:53'] servers UP
2025-12-08 04:17:22,161 - INFO - an-jenkins-deploy-platform-ansible-os-4854-1 connected to dns 10.20.0.14
2025-12-08 04:17:22,161 - WARNING - an-jenkins-deploy-platform-ansible-os-4854-1:contrail-vrouter-agent module connection to ['10.20.0.145:53'] servers NOT UP
2025-12-08 04:17:22,161 - WARNING - an-jenkins-deploy-platform-ansible-os-4854-1:contrail-vrouter-agent module connection to ['10.0.0.23:8086'] servers NOT UP
2025-12-08 04:17:22,161 - WARNING - an-jenkins-deploy-platform-ansible-os-4854-1:contrail-vrouter-agent module connection to ['10.0.0.22:8086'] servers NOT UP
2025-12-08 04:17:22,161 - INFO - an-jenkins-deploy-platform-ansible-os-4854-1:contrail-vrouter-agent module connection to ['10.0.0.65:8086'] servers UP
2025-12-08 04:17:22,161 - INFO - an-jenkins-deploy-platform-ansible-os-4854-1 connected to collector 10.0.0.65
2025-12-08 04:17:22,161 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:17:22,655 - INFO - cn-jenkins-deploy-platform-ansible-os-4854-1:contrail-control module connection to 10.0.0.23:9041 servers UP
2025-12-08 04:17:22,655 - INFO - cn-jenkins-deploy-platform-ansible-os-4854-1:contrail-control module connection to 10.0.0.23:5673 servers UP
2025-12-08 04:17:22,656 - INFO - cn-jenkins-deploy-platform-ansible-os-4854-1:contrail-control module connection to ['10.0.0.23:8086'] servers UP
2025-12-08 04:17:22,656 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4854-2?flat
2025-12-08 04:17:23,104 - INFO - cn-jenkins-deploy-platform-ansible-os-4854-2:contrail-control module connection to 10.0.0.23:9041 servers UP
2025-12-08 04:17:23,104 - INFO - cn-jenkins-deploy-platform-ansible-os-4854-2:contrail-control module connection to 10.0.0.23:5673 servers UP
2025-12-08 04:17:23,105 - INFO - cn-jenkins-deploy-platform-ansible-os-4854-2:contrail-control module connection to ['10.0.0.23:8086'] servers UP
2025-12-08 04:17:23,105 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4854-3?flat
2025-12-08 04:17:23,125 - INFO - cn-jenkins-deploy-platform-ansible-os-4854-3:contrail-control module connection to 10.0.0.23:9041 servers UP
2025-12-08 04:17:23,125 - INFO - cn-jenkins-deploy-platform-ansible-os-4854-3:contrail-control module connection to 10.0.0.23:5673 servers UP
2025-12-08 04:17:23,126 - INFO - cn-jenkins-deploy-platform-ansible-os-4854-3:contrail-control module connection to ['10.0.0.23:8086'] servers UP
2025-12-08 04:17:23,126 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4854-1.?flat
2025-12-08 04:17:23,152 - DEBUG - contrail-api is running
2025-12-08 04:17:23,152 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4854-2.?flat
2025-12-08 04:17:23,185 - DEBUG - contrail-api is running
2025-12-08 04:17:23,185 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4854-3.?flat
2025-12-08 04:17:23,213 - DEBUG - contrail-api is running
2025-12-08 04:17:23,533 - INFO - Created VN ctest-vn3-32268234, UUID :c8989105-6027-4490-9117-33b38c5796d0
2025-12-08 04:17:23,973 - DEBUG - Requesting: http://10.0.0.23:8082/domains
2025-12-08 04:17:24,007 - DEBUG - Requesting: http://10.0.0.23:8082/domain/c5befdc1-279e-4ac1-9620-9608231491f8
2025-12-08 04:17:24,020 - DEBUG - Requesting: http://10.0.0.23:8082/project/39b9363a-5400-4ea1-a180-7dd8e2fe11b9
2025-12-08 04:17:24,116 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/75a3447b-83a9-421f-b144-cf08e0cc11a5
2025-12-08 04:17:24,135 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/75a3447b-83a9-421f-b144-cf08e0cc11a5
2025-12-08 04:17:24,149 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/904d4940-9135-42bf-812f-06d264925cbe
2025-12-08 04:17:24,157 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/904d4940-9135-42bf-812f-06d264925cbe
2025-12-08 04:17:24,185 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/3ddfeaf6-800b-4608-a01a-f15599d0b6a6
2025-12-08 04:17:24,194 - DEBUG - Route Targets: ['target:64512:8000004']
2025-12-08 04:17:24,194 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/75a3447b-83a9-421f-b144-cf08e0cc11a5
2025-12-08 04:17:24,204 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/904d4940-9135-42bf-812f-06d264925cbe
2025-12-08 04:17:24,226 - INFO - Verified VN network id 14 for VN 75a3447b-83a9-421f-b144-cf08e0cc11a5
2025-12-08 04:17:24,226 - INFO - Verifications in API Server for VN ctest-vn1-11209066 passed
2025-12-08 04:17:24,226 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/75a3447b-83a9-421f-b144-cf08e0cc11a5
2025-12-08 04:17:24,236 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/904d4940-9135-42bf-812f-06d264925cbe
2025-12-08 04:17:24,245 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/904d4940-9135-42bf-812f-06d264925cbe
2025-12-08 04:17:24,255 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/3ddfeaf6-800b-4608-a01a-f15599d0b6a6
2025-12-08 04:17:24,274 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'interests': '1', 'advertised': '1', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-32026406', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '8476694220921979423', 'uuid-lslong': '12773562080167727525'}, 'enable': 'true', 'created': '2025-12-08T04:14:24', 'last-modified': '2025-12-08T04:14:25', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:07.166659'}
2025-12-08 04:17:24,294 - DEBUG - Route Targets: ['target:64512:8000004']
2025-12-08 04:17:24,302 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-32026406', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '8476694220921979423', 'uuid-lslong': '12773562080167727525'}, 'enable': 'true', 'created': '2025-12-08T04:14:24', 'last-modified': '2025-12-08T04:14:25', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:01.811227'}
2025-12-08 04:17:24,308 - DEBUG - Route Targets: ['target:64512:8000004']
2025-12-08 04:17:24,315 - DEBUG - Control-node 10.0.0.65 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-32026406', 'owner-access': '7', 'group': 'member', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '8476694220921979423', 'uuid-lslong': '12773562080167727525'}, 'enable': 'true', 'created': '2025-12-08T04:14:24', 'last-modified': '2025-12-08T04:14:25', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:56.304228'}
2025-12-08 04:17:24,322 - DEBUG - Route Targets: ['target:64512:8000004']
2025-12-08 04:17:24,322 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-11209066 passed
2025-12-08 04:17:24,322 - DEBUG - ====Verifying policy data for ctest-vn1-11209066 in API_Server ======
2025-12-08 04:17:24,323 - DEBUG - Requesting: http://10.0.0.23:8082/domains
2025-12-08 04:17:24,330 - DEBUG - Requesting: http://10.0.0.23:8082/domain/c5befdc1-279e-4ac1-9620-9608231491f8
2025-12-08 04:17:24,346 - DEBUG - Requesting: http://10.0.0.23:8082/project/39b9363a-5400-4ea1-a180-7dd8e2fe11b9
2025-12-08 04:17:24,446 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/75a3447b-83a9-421f-b144-cf08e0cc11a5
2025-12-08 04:17:24,456 - DEBUG - ==>Verifying data for policy with id: b31e551b-0051-454a-8565-2ab3a7c8cfdd, fqn: ['default-domain', 'ctest-TestBasicVMVN0-32026406', 'policy1']
2025-12-08 04:17:24,456 - INFO - VN ctest-vn1-11209066 Policy verification: verify_vn_policy_in_api_server, status: True
2025-12-08 04:17:24,456 - DEBUG - Verifying the vn in opserver
2025-12-08 04:17:24,457 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 virtual network link through opserver 10.0.0.23
2025-12-08 04:17:24,457 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks
2025-12-08 04:17:24,470 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066?flat'}
2025-12-08 04:17:24,470 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 virtual network link through opserver 10.0.0.22
2025-12-08 04:17:24,470 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks
2025-12-08 04:17:24,541 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066?flat'}
2025-12-08 04:17:24,541 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 virtual network link through opserver 10.0.0.65
2025-12-08 04:17:24,541 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-networks
2025-12-08 04:17:24,610 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'href': 'http://10.0.0.65:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066?flat'}
2025-12-08 04:17:24,610 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is found in opserver
2025-12-08 04:17:24,623 - DEBUG - VRF ids for VN ctest-vn1-11209066: {'10.0.0.129': '2'}
2025-12-08 04:17:26,032 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4854-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4854-2)
2025-12-08 04:17:26,096 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2025-12-08 04:17:31,182 - DEBUG - VM is in ACTIVE state now
2025-12-08 04:17:31,182 - INFO - VM name : ctest-vn1_vm2-50808408
2025-12-08 04:17:31,274 - DEBUG - VM ctest-vn1_vm2-50808408 ID is b59971a0-b5d8-4726-a009-3db7c055d128
2025-12-08 04:17:31,354 - DEBUG - VM is in ACTIVE state now
2025-12-08 04:17:31,355 - DEBUG - VM ctest-vn1_vm2-50808408 launched on Node an-jenkins-deploy-platform-ansible-os-4854-2
2025-12-08 04:17:31,438 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128
2025-12-08 04:17:31,479 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128
2025-12-08 04:17:31,487 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/5d282731-8acb-4ac6-bf43-e58e7af6552b
2025-12-08 04:17:31,497 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128
2025-12-08 04:17:31,504 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/5d282731-8acb-4ac6-bf43-e58e7af6552b
2025-12-08 04:17:31,512 - DEBUG - Requesting: http://10.0.0.65:8082/instance-ip/6d5d2a4b-afe1-4ff0-9f01-254463e5ef88
2025-12-08 04:17:31,521 - DEBUG - Verifying in api server 10.0.0.65
2025-12-08 04:17:31,521 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128
2025-12-08 04:17:31,528 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/5d282731-8acb-4ac6-bf43-e58e7af6552b
2025-12-08 04:17:31,536 - DEBUG - Requesting: http://10.0.0.65:8082/instance-ip/6d5d2a4b-afe1-4ff0-9f01-254463e5ef88
2025-12-08 04:17:31,543 - INFO - VM ctest-vn1_vm2-50808408 verfication in all API Servers passed
2025-12-08 04:17:31,563 - DEBUG - VM ctest-vn1_vm2-50808408 Tap interface: {'index': '4', 'name': 'tap5d282731-8a', 'uuid': '5d282731-8acb-4ac6-bf43-e58e7af6552b', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066:ctest-vn1-11209066', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'vm_uuid': 'b59971a0-b5d8-4726-a009-3db7c055d128', 'vm_name': 'ctest-vn1_vm2-50808408', 'ip_addr': '192.168.1.251', 'mac_addr': '02:5d:28:27:31:8a', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '169', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-32026406:5d282731-8acb-4ac6-bf43-e58e7af6552b', 'sg_uuid_list': ['a08ad7f7-4fb9-4642-b81a-e5edfb06e45f'], 'static_route_list': None, 'vm_project_uuid': '39b9363a-5400-4ea1-a180-7dd8e2fe11b9', 'admin_state': 'Enabled', 'flow_key_idx': '22', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-12-08 04:17:31,563 - DEBUG - Agent 10.0.0.129 vrf name: default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066:ctest-vn1-11209066
2025-12-08 04:17:31,575 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066:ctest-vn1-11209066', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'table_label': '-1', 'vxlan_id': '14', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.19:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]}
2025-12-08 04:17:31,655 - DEBUG - VM ctest-vn1_vm2-50808408 : Tap interface tap5d282731-8a is set to Active
2025-12-08 04:17:31,656 - DEBUG - Tap interface tap5d282731-8a detail : {'index': '4', 'name': 'tap5d282731-8a', 'uuid': '5d282731-8acb-4ac6-bf43-e58e7af6552b', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066:ctest-vn1-11209066', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '14', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'vm_uuid': 'b59971a0-b5d8-4726-a009-3db7c055d128', 'vm_name': 'ctest-vn1_vm2-50808408', 'ip_addr': '192.168.1.251', 'mac_addr': '02:5d:28:27:31:8a', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '169', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-32026406:5d282731-8acb-4ac6-bf43-e58e7af6552b', 'sg_uuid_list': ['a08ad7f7-4fb9-4642-b81a-e5edfb06e45f'], 'static_route_list': None, 'vm_project_uuid': '39b9363a-5400-4ea1-a180-7dd8e2fe11b9', 'admin_state': 'Enabled', 'flow_key_idx': '22', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-12-08 04:17:31,768 - DEBUG - Starting Layer 2 verification in Agent
2025-12-08 04:17:31,783 - DEBUG - Layer 2 path is seen for VM MAC 02:5d:28:27:31:8a in agent 10.0.0.129
2025-12-08 04:17:31,784 - DEBUG - Active layer 2 route in agent is present for VMI tap5d282731-8a
2025-12-08 04:17:31,784 - DEBUG - L2 label(34) matches bw route table and interface table
2025-12-08 04:17:31,797 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is found in Agent of node 10.0.0.129
2025-12-08 04:17:31,932 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is consistent in agent 10.0.0.129
2025-12-08 04:17:31,932 - DEBUG - Route for VM IP ['192.168.1.251'] is consistent in agent 10.0.0.129
2025-12-08 04:17:31,932 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 verification for VM ctest-vn1_vm2-50808408 in Agent 10.0.0.129 passed
2025-12-08 04:17:32,046 - DEBUG - Starting all layer 2 verification in agent 10.0.0.129
2025-12-08 04:17:32,060 - DEBUG - Route for VM MAC 02:5d:28:27:31:8a is consistent in agent 10.0.0.129
2025-12-08 04:17:35,267 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1020ms')
2025-12-08 04:17:35,267 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-50808408 failed!
2025-12-08 04:17:35,280 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:17:39,361 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms')
2025-12-08 04:17:39,361 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-50808408 failed!
2025-12-08 04:17:39,376 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:17:43,455 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms')
2025-12-08 04:17:43,455 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-50808408 failed!
2025-12-08 04:17:43,469 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:17:47,546 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms')
2025-12-08 04:17:47,546 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-50808408 failed!
2025-12-08 04:17:47,559 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:17:51,654 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1029ms')
2025-12-08 04:17:51,654 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-50808408 failed!
2025-12-08 04:17:51,669 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:17:55,745 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms')
2025-12-08 04:17:55,745 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-50808408 failed!
2025-12-08 04:17:55,758 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:17:59,836 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms')
2025-12-08 04:17:59,836 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-50808408 failed!
2025-12-08 04:17:59,849 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:18:03,930 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms')
2025-12-08 04:18:03,930 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-50808408 failed!
2025-12-08 04:18:03,944 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:18:08,031 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms')
2025-12-08 04:18:08,031 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-50808408 failed!
2025-12-08 04:18:08,046 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 is 192.168.1.254 and allocation pool is NOT set
2025-12-08 04:18:10,111 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=6.31 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.693 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.693/3.501/6.310/2.808 ms')
2025-12-08 04:18:10,112 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-50808408 passed
2025-12-08 04:18:10,112 - INFO - VM ctest-vn1_vm2-50808408 verifications in Compute nodes passed
2025-12-08 04:18:10,331 - DEBUG - Validated VM route 192.168.1.251 in vrouter of 10.0.0.129
2025-12-08 04:18:10,336 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0
2025-12-08 04:18:10,336 - INFO - Validated routes of VM ctest-vn1_vm2-50808408 in all vrouters
2025-12-08 04:18:10,753 - DEBUG - Starting all layer2 verification in 10.0.0.65 Control Node
2025-12-08 04:18:10,878 - DEBUG - Layer2 route found for VM MAC 02:5d:28:27:31:8a in Control-node 10.0.0.65
2025-12-08 04:18:10,878 - DEBUG - L2 Label for VM ctest-vn1_vm2-50808408 same between Control-node 10.0.0.65 and Agent, Expected: 34, Seen: 34
2025-12-08 04:18:10,878 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node
2025-12-08 04:18:11,007 - DEBUG - Layer2 route found for VM MAC 02:5d:28:27:31:8a in Control-node 10.0.0.23
2025-12-08 04:18:11,007 - DEBUG - L2 Label for VM ctest-vn1_vm2-50808408 same between Control-node 10.0.0.23 and Agent, Expected: 34, Seen: 34
2025-12-08 04:18:11,007 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-50808408 passed
2025-12-08 04:18:11,007 - DEBUG - Verifying the vm in opserver
2025-12-08 04:18:11,007 - DEBUG - Verifying in collector 10.0.0.65 ...
2025-12-08 04:18:11,007 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128?flat
2025-12-08 04:18:11,075 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-32026406:5d282731-8acb-4ac6-bf43-e58e7af6552b?flat
2025-12-08 04:18:11,100 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-32026406:5d282731-8acb-4ac6-bf43-e58e7af6552b?flat
2025-12-08 04:18:11,122 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128?flat
2025-12-08 04:18:11,189 - DEBUG - Verifying vm in vn uve
2025-12-08 04:18:11,190 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-32026406:5d282731-8acb-4ac6-bf43-e58e7af6552b?flat
2025-12-08 04:18:11,211 - DEBUG - VM uve shows interface as {'ip_address': '192.168.1.251', 'mac_address': '02:5d:28:27:31:8a', 'vm_name': 'ctest-vn1_vm2-50808408', 'admin_state': True, 'ip6_address': '::', 'rx_vlan': 65535, 'label': 30, 'gateway': '192.168.1.254', 'vm_uuid': 'b59971a0-b5d8-4726-a009-3db7c055d128', 'is_health_check_active': True, 'active': True, 'uuid': '5d282731-8acb-4ac6-bf43-e58e7af6552b', '__T': 1765167461628349, 'ip6_active': False, 'ip4_active': True, 'l2_active': True, 'port_mirror_enabled': False, 'vhostuser_mode': 0, 'fixed_ip4_list': ['192.168.1.251'], 'tx_vlan': 65535, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066', 'vn_uuid': '75a3447b-83a9-421f-b144-cf08e0cc11a5'}
2025-12-08 04:18:11,211 - DEBUG - VM uve shows ip address as ['192.168.1.251', '::']
2025-12-08 04:18:11,211 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066
2025-12-08 04:18:11,211 - INFO - Verifying through opserver in 10.0.0.23
2025-12-08 04:18:11,211 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066?flat
2025-12-08 04:18:11,237 - DEBUG - expected vm list b59971a0-b5d8-4726-a009-3db7c055d128
2025-12-08 04:18:11,238 - DEBUG - Extracted vm list ['b59971a0-b5d8-4726-a009-3db7c055d128']
2025-12-08 04:18:11,238 - INFO - VM b59971a0-b5d8-4726-a009-3db7c055d128 is present in default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066
2025-12-08 04:18:11,238 - DEBUG - Verifying vm in vrouter uve
2025-12-08 04:18:11,238 - DEBUG - Getting info from collector 10.0.0.65..
2025-12-08 04:18:11,238 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128?flat
2025-12-08 04:18:11,255 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4854-2']
2025-12-08 04:18:11,256 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-2:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:18:11,310 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-2?flat
2025-12-08 04:18:11,346 - DEBUG - VM b59971a0-b5d8-4726-a009-3db7c055d128 is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-2 uve
2025-12-08 04:18:11,346 - DEBUG - Validated that VM ctest-vn1_vm2-50808408 is in Vrouter an-jenkins-deploy-platform-ansible-os-4854-2 UVE
2025-12-08 04:18:11,346 - DEBUG - Verifying vm tap interface/vn in vrouter uve
2025-12-08 04:18:11,355 - DEBUG - Expected tap interface of VM uuid b59971a0-b5d8-4726-a009-3db7c055d128 is default-domain:ctest-TestBasicVMVN0-32026406:5d282731-8acb-4ac6-bf43-e58e7af6552b
2025-12-08 04:18:11,355 - DEBUG - Expected VN of VM uuid b59971a0-b5d8-4726-a009-3db7c055d128 is default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066
2025-12-08 04:18:11,355 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-2:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:18:11,406 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-2?flat
2025-12-08 04:18:11,440 - DEBUG - VM b59971a0-b5d8-4726-a009-3db7c055d128 is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-2 uve
2025-12-08 04:18:11,440 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-32026406:5d282731-8acb-4ac6-bf43-e58e7af6552b of vm b59971a0-b5d8-4726-a009-3db7c055d128 is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-2 uve
2025-12-08 04:18:11,440 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 of vm b59971a0-b5d8-4726-a009-3db7c055d128 is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-2 uve
2025-12-08 04:18:11,440 - DEBUG - Verifying the b59971a0-b5d8-4726-a009-3db7c055d128 virtual network link through opserver 10.0.0.23
2025-12-08 04:18:11,440 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines
2025-12-08 04:18:11,454 - DEBUG - VM link and name as {'name': 'b59971a0-b5d8-4726-a009-3db7c055d128', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128?flat'}
2025-12-08 04:18:11,454 - DEBUG - Verifying the b59971a0-b5d8-4726-a009-3db7c055d128 virtual network link through opserver 10.0.0.22
2025-12-08 04:18:11,454 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-12-08 04:18:11,520 - DEBUG - VM link and name as {'name': 'b59971a0-b5d8-4726-a009-3db7c055d128', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128?flat'}
2025-12-08 04:18:11,521 - DEBUG - Verifying the b59971a0-b5d8-4726-a009-3db7c055d128 virtual network link through opserver 10.0.0.65
2025-12-08 04:18:11,521 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machines
2025-12-08 04:18:11,534 - DEBUG - VM link and name as {'name': 'b59971a0-b5d8-4726-a009-3db7c055d128', 'href': 'http://10.0.0.65:8081/analytics/uves/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128?flat'}
2025-12-08 04:18:11,534 - INFO - VM ctest-vn1_vm2-50808408 validations in Opserver passed
2025-12-08 04:18:12,848 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4854-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4854-1)
2025-12-08 04:18:12,902 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2025-12-08 04:18:17,980 - DEBUG - VM is in ACTIVE state now
2025-12-08 04:18:17,980 - INFO - VM name : ctest-vn2_vm2-18290950
2025-12-08 04:18:18,079 - DEBUG - VM ctest-vn2_vm2-18290950 ID is a9733ad1-0267-4d1d-9de5-77b8687a88bd
2025-12-08 04:18:18,163 - DEBUG - VM is in ACTIVE state now
2025-12-08 04:18:18,163 - DEBUG - VM ctest-vn2_vm2-18290950 launched on Node an-jenkins-deploy-platform-ansible-os-4854-1
2025-12-08 04:18:18,248 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd
2025-12-08 04:18:18,260 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd
2025-12-08 04:18:18,269 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/ec393981-9dbc-4828-934a-6974dce0fd35
2025-12-08 04:18:18,280 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd
2025-12-08 04:18:18,288 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/ec393981-9dbc-4828-934a-6974dce0fd35
2025-12-08 04:18:18,299 - DEBUG - Requesting: http://10.0.0.65:8082/instance-ip/d77ea983-f17c-49ce-8da8-91aba1d40826
2025-12-08 04:18:18,310 - DEBUG - Verifying in api server 10.0.0.65
2025-12-08 04:18:18,311 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd
2025-12-08 04:18:18,319 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/ec393981-9dbc-4828-934a-6974dce0fd35
2025-12-08 04:18:18,329 - DEBUG - Requesting: http://10.0.0.65:8082/instance-ip/d77ea983-f17c-49ce-8da8-91aba1d40826
2025-12-08 04:18:18,339 - INFO - VM ctest-vn2_vm2-18290950 verfication in all API Servers passed
2025-12-08 04:18:18,359 - DEBUG - VM ctest-vn2_vm2-18290950 Tap interface: {'index': '4', 'name': 'tapec393981-9d', 'uuid': 'ec393981-9dbc-4828-934a-6974dce0fd35', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834:ctest-vn2-01263834', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834', 'vm_uuid': 'a9733ad1-0267-4d1d-9de5-77b8687a88bd', 'vm_name': 'ctest-vn2_vm2-18290950', 'ip_addr': '192.168.2.251', 'mac_addr': '02:ec:39:39:81:9d', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ec393981-9dbc-4828-934a-6974dce0fd35', 'sg_uuid_list': ['a08ad7f7-4fb9-4642-b81a-e5edfb06e45f'], 'static_route_list': None, 'vm_project_uuid': '39b9363a-5400-4ea1-a180-7dd8e2fe11b9', 'admin_state': 'Enabled', 'flow_key_idx': '19', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-12-08 04:18:18,359 - DEBUG - Agent 10.0.0.41 vrf name: default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834:ctest-vn2-01263834
2025-12-08 04:18:18,365 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834:ctest-vn2-01263834', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834', 'table_label': '-1', 'vxlan_id': '15', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.252:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]}
2025-12-08 04:18:18,447 - DEBUG - VM ctest-vn2_vm2-18290950 : Tap interface tapec393981-9d is set to Active
2025-12-08 04:18:18,448 - DEBUG - Tap interface tapec393981-9d detail : {'index': '4', 'name': 'tapec393981-9d', 'uuid': 'ec393981-9dbc-4828-934a-6974dce0fd35', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834:ctest-vn2-01263834', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '15', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834', 'vm_uuid': 'a9733ad1-0267-4d1d-9de5-77b8687a88bd', 'vm_name': 'ctest-vn2_vm2-18290950', 'ip_addr': '192.168.2.251', 'mac_addr': '02:ec:39:39:81:9d', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '127', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-32026406:ec393981-9dbc-4828-934a-6974dce0fd35', 'sg_uuid_list': ['a08ad7f7-4fb9-4642-b81a-e5edfb06e45f'], 'static_route_list': None, 'vm_project_uuid': '39b9363a-5400-4ea1-a180-7dd8e2fe11b9', 'admin_state': 'Enabled', 'flow_key_idx': '19', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-12-08 04:18:18,561 - DEBUG - Starting Layer 2 verification in Agent
2025-12-08 04:18:18,576 - DEBUG - Layer 2 path is seen for VM MAC 02:ec:39:39:81:9d in agent 10.0.0.41
2025-12-08 04:18:18,576 - DEBUG - Active layer 2 route in agent is present for VMI tapec393981-9d
2025-12-08 04:18:18,576 - DEBUG - L2 label(34) matches bw route table and interface table
2025-12-08 04:18:18,584 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 is found in Agent of node 10.0.0.41
2025-12-08 04:18:18,718 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 is consistent in agent 10.0.0.41
2025-12-08 04:18:18,719 - DEBUG - Route for VM IP ['192.168.2.251'] is consistent in agent 10.0.0.41
2025-12-08 04:18:18,719 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 verification for VM ctest-vn2_vm2-18290950 in Agent 10.0.0.41 passed
2025-12-08 04:18:18,838 - DEBUG - Starting all layer 2 verification in agent 10.0.0.41
2025-12-08 04:18:18,853 - DEBUG - Route for VM MAC 02:ec:39:39:81:9d is consistent in agent 10.0.0.41
2025-12-08 04:18:22,061 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1027ms')
2025-12-08 04:18:22,061 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-18290950 failed!
2025-12-08 04:18:22,074 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 is 192.168.2.254 and allocation pool is NOT set
2025-12-08 04:18:26,158 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms')
2025-12-08 04:18:26,158 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-18290950 failed!
2025-12-08 04:18:26,173 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 is 192.168.2.254 and allocation pool is NOT set
2025-12-08 04:18:30,262 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms')
2025-12-08 04:18:30,262 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-18290950 failed!
2025-12-08 04:18:30,276 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 is 192.168.2.254 and allocation pool is NOT set
2025-12-08 04:18:34,352 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms')
2025-12-08 04:18:34,352 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-18290950 failed!
2025-12-08 04:18:34,367 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 is 192.168.2.254 and allocation pool is NOT set
2025-12-08 04:18:38,442 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms')
2025-12-08 04:18:38,442 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-18290950 failed!
2025-12-08 04:18:38,456 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 is 192.168.2.254 and allocation pool is NOT set
2025-12-08 04:18:42,544 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms')
2025-12-08 04:18:42,545 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-18290950 failed!
2025-12-08 04:18:42,559 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 is 192.168.2.254 and allocation pool is NOT set
2025-12-08 04:18:46,642 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms')
2025-12-08 04:18:46,642 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-18290950 failed!
2025-12-08 04:18:46,656 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 is 192.168.2.254 and allocation pool is NOT set
2025-12-08 04:18:50,733 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms')
2025-12-08 04:18:50,733 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-18290950 failed!
2025-12-08 04:18:50,751 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 is 192.168.2.254 and allocation pool is NOT set
2025-12-08 04:18:52,822 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=13.2 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=2.99 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 2.986/8.084/13.183/5.098 ms')
2025-12-08 04:18:52,822 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-18290950 passed
2025-12-08 04:18:52,822 - INFO - VM ctest-vn2_vm2-18290950 verifications in Compute nodes passed
2025-12-08 04:18:53,057 - DEBUG - Validated VM route 192.168.2.251 in vrouter of 10.0.0.41
2025-12-08 04:18:53,062 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0
2025-12-08 04:18:53,062 - INFO - Validated routes of VM ctest-vn2_vm2-18290950 in all vrouters
2025-12-08 04:18:53,483 - DEBUG - Starting all layer2 verification in 10.0.0.65 Control Node
2025-12-08 04:18:53,603 - DEBUG - Layer2 route found for VM MAC 02:ec:39:39:81:9d in Control-node 10.0.0.65
2025-12-08 04:18:53,603 - DEBUG - L2 Label for VM ctest-vn2_vm2-18290950 same between Control-node 10.0.0.65 and Agent, Expected: 34, Seen: 34
2025-12-08 04:18:53,603 - DEBUG - Starting all layer2 verification in 10.0.0.23 Control Node
2025-12-08 04:18:53,725 - DEBUG - Layer2 route found for VM MAC 02:ec:39:39:81:9d in Control-node 10.0.0.23
2025-12-08 04:18:53,725 - DEBUG - L2 Label for VM ctest-vn2_vm2-18290950 same between Control-node 10.0.0.23 and Agent, Expected: 34, Seen: 34
2025-12-08 04:18:53,725 - INFO - Verification in Control-nodes for VM ctest-vn2_vm2-18290950 passed
2025-12-08 04:18:53,725 - DEBUG - Verifying the vm in opserver
2025-12-08 04:18:53,725 - DEBUG - Verifying in collector 10.0.0.65 ...
2025-12-08 04:18:53,725 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd?flat
2025-12-08 04:18:53,740 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-32026406:ec393981-9dbc-4828-934a-6974dce0fd35?flat
2025-12-08 04:18:53,759 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-32026406:ec393981-9dbc-4828-934a-6974dce0fd35?flat
2025-12-08 04:18:53,778 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd?flat
2025-12-08 04:18:53,792 - DEBUG - Verifying vm in vn uve
2025-12-08 04:18:53,792 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-32026406:ec393981-9dbc-4828-934a-6974dce0fd35?flat
2025-12-08 04:18:53,809 - DEBUG - VM uve shows interface as {'rx_vlan': 65535, 'admin_state': True, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834', 'l2_active': True, '__T': 1765167513043763, 'vm_uuid': 'a9733ad1-0267-4d1d-9de5-77b8687a88bd', 'is_health_check_active': True, 'active': True, 'ip6_address': '::', 'vhostuser_mode': 0, 'ip4_active': True, 'vm_name': 'ctest-vn2_vm2-18290950', 'fixed_ip4_list': ['192.168.2.251'], 'tx_vlan': 65535, 'vn_uuid': '209f34d1-dfa3-4731-a388-f9ca87d11aad', 'ip_address': '192.168.2.251', 'label': 30, 'gateway': '192.168.2.254', 'uuid': 'ec393981-9dbc-4828-934a-6974dce0fd35', 'mac_address': '02:ec:39:39:81:9d', 'ip6_active': False, 'port_mirror_enabled': False}
2025-12-08 04:18:53,809 - DEBUG - VM uve shows ip address as ['192.168.2.251', '::']
2025-12-08 04:18:53,809 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834
2025-12-08 04:18:53,809 - INFO - Verifying through opserver in 10.0.0.23
2025-12-08 04:18:53,809 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834?flat
2025-12-08 04:18:53,834 - DEBUG - expected vm list a9733ad1-0267-4d1d-9de5-77b8687a88bd
2025-12-08 04:18:53,834 - DEBUG - Extracted vm list ['a9733ad1-0267-4d1d-9de5-77b8687a88bd']
2025-12-08 04:18:53,834 - INFO - VM a9733ad1-0267-4d1d-9de5-77b8687a88bd is present in default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834
2025-12-08 04:18:53,834 - DEBUG - Verifying vm in vrouter uve
2025-12-08 04:18:53,834 - DEBUG - Getting info from collector 10.0.0.65..
2025-12-08 04:18:53,834 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd?flat
2025-12-08 04:18:53,848 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4854-1']
2025-12-08 04:18:53,848 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:18:53,908 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:18:53,937 - DEBUG - VM a9733ad1-0267-4d1d-9de5-77b8687a88bd is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:18:53,937 - DEBUG - Validated that VM ctest-vn2_vm2-18290950 is in Vrouter an-jenkins-deploy-platform-ansible-os-4854-1 UVE
2025-12-08 04:18:53,937 - DEBUG - Verifying vm tap interface/vn in vrouter uve
2025-12-08 04:18:53,945 - DEBUG - Expected tap interface of VM uuid a9733ad1-0267-4d1d-9de5-77b8687a88bd is default-domain:ctest-TestBasicVMVN0-32026406:ec393981-9dbc-4828-934a-6974dce0fd35
2025-12-08 04:18:53,945 - DEBUG - Expected VN of VM uuid a9733ad1-0267-4d1d-9de5-77b8687a88bd is default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834
2025-12-08 04:18:53,945 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:18:54,003 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:18:54,031 - DEBUG - VM a9733ad1-0267-4d1d-9de5-77b8687a88bd is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:18:54,031 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-32026406:ec393981-9dbc-4828-934a-6974dce0fd35 of vm a9733ad1-0267-4d1d-9de5-77b8687a88bd is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:18:54,031 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 of vm a9733ad1-0267-4d1d-9de5-77b8687a88bd is present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:18:54,031 - DEBUG - Verifying the a9733ad1-0267-4d1d-9de5-77b8687a88bd virtual network link through opserver 10.0.0.23
2025-12-08 04:18:54,031 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines
2025-12-08 04:18:54,043 - DEBUG - VM link and name as {'name': 'a9733ad1-0267-4d1d-9de5-77b8687a88bd', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd?flat'}
2025-12-08 04:18:54,043 - DEBUG - Verifying the a9733ad1-0267-4d1d-9de5-77b8687a88bd virtual network link through opserver 10.0.0.22
2025-12-08 04:18:54,043 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-12-08 04:18:54,056 - DEBUG - VM link and name as {'name': 'a9733ad1-0267-4d1d-9de5-77b8687a88bd', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd?flat'}
2025-12-08 04:18:54,056 - DEBUG - Verifying the a9733ad1-0267-4d1d-9de5-77b8687a88bd virtual network link through opserver 10.0.0.65
2025-12-08 04:18:54,056 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machines
2025-12-08 04:18:54,067 - DEBUG - VM link and name as {'name': 'a9733ad1-0267-4d1d-9de5-77b8687a88bd', 'href': 'http://10.0.0.65:8081/analytics/uves/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd?flat'}
2025-12-08 04:18:54,067 - INFO - VM ctest-vn2_vm2-18290950 validations in Opserver passed
2025-12-08 04:18:54,067 - INFO - Waiting for VM ctest-vn1_vm2-50808408 to be up..
2025-12-08 04:18:54,068 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:18:54,068 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128
2025-12-08 04:18:54,075 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/5d282731-8acb-4ac6-bf43-e58e7af6552b
2025-12-08 04:18:55,231 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.12 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.495 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1000ms\r\nrtt min/avg/max/mdev = 0.495/1.808/3.122/1.313 ms')
2025-12-08 04:18:55,231 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-50808408 passed
2025-12-08 04:18:55,297 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:18:55,298 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-50808408, IP 192.168.1.251, Port 22
2025-12-08 04:18:55,466 - DEBUG - VM ctest-vn1_vm2-50808408 is ready for SSH connections
2025-12-08 04:18:55,466 - INFO - Waiting for VM ctest-vn2_vm2-18290950 to be up..
2025-12-08 04:18:55,466 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:18:55,466 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd
2025-12-08 04:18:55,475 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine-interface/ec393981-9dbc-4828-934a-6974dce0fd35
2025-12-08 04:18:56,634 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=1.37 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.377 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.377/0.871/1.366/0.494 ms')
2025-12-08 04:18:56,634 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-18290950 passed
2025-12-08 04:18:56,701 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:18:56,701 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-18290950, IP 192.168.2.251, Port 22
2025-12-08 04:18:56,769 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-12-08 04:18:56,857 - DEBUG - VM ctest-vn2_vm2-18290950 is NOT ready for SSH connections, VM status: ACTIVE
2025-12-08 04:19:01,857 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:19:01,857 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-18290950, IP 192.168.2.251, Port 22
2025-12-08 04:19:01,925 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-12-08 04:19:02,008 - DEBUG - VM ctest-vn2_vm2-18290950 is NOT ready for SSH connections, VM status: ACTIVE
2025-12-08 04:19:07,009 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:19:07,010 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-18290950, IP 192.168.2.251, Port 22
2025-12-08 04:19:07,078 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-12-08 04:19:07,169 - DEBUG - VM ctest-vn2_vm2-18290950 is NOT ready for SSH connections, VM status: ACTIVE
2025-12-08 04:19:12,169 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:19:12,170 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-18290950, IP 192.168.2.251, Port 22
2025-12-08 04:19:12,327 - DEBUG - VM ctest-vn2_vm2-18290950 is ready for SSH connections
2025-12-08 04:19:12,327 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-12-08 04:19:12,327 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.251, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.129, gateway password: c0ntrail123
2025-12-08 04:19:12,327 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.251
2025-12-08 04:19:53,199 - DEBUG - PING 192.168.2.251 (192.168.2.251) 56(84) bytes of data.
64 bytes from 192.168.2.251: icmp_req=1 ttl=63 time=10.1 ms
64 bytes from 192.168.2.251: icmp_req=2 ttl=63 time=1.17 ms
64 bytes from 192.168.2.251: icmp_req=3 ttl=63 time=2.11 ms
--- 192.168.2.251 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.175/4.494/10.194/4.049 ms
2025-12-08 04:19:53,200 - INFO - Ping to IP 192.168.2.251 from VM ctest-vn1_vm2-50808408 passed
2025-12-08 04:19:53,200 - INFO - Deleting VM ctest-vn2_vm2-18290950
2025-12-08 04:19:53,290 - DEBUG - Verifying in api server 10.0.0.65
2025-12-08 04:19:53,290 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd
2025-12-08 04:19:53,298 - DEBUG - VM ID a9733ad1-0267-4d1d-9de5-77b8687a88bd of VM ctest-vn2_vm2-18290950 is still found in API Server
2025-12-08 04:19:55,300 - DEBUG - Verifying in api server 10.0.0.65
2025-12-08 04:19:55,300 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd
2025-12-08 04:19:55,307 - DEBUG - Response Code: 404
2025-12-08 04:19:55,308 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd
2025-12-08 04:19:55,315 - DEBUG - Response Code: 404
2025-12-08 04:19:55,315 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/a9733ad1-0267-4d1d-9de5-77b8687a88bd
2025-12-08 04:19:55,322 - DEBUG - Response Code: 404
2025-12-08 04:19:55,322 - INFO - VM ctest-vn2_vm2-18290950 is fully removed in API-Server
2025-12-08 04:19:55,473 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found
2025-12-08 04:19:55,656 - DEBUG - VM route 192.168.2.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 of compute 10.0.0.41
2025-12-08 04:19:55,661 - DEBUG - Validated that vrouter 10.0.0.41 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834
2025-12-08 04:19:55,661 - DEBUG - Validated that vrouter 10.0.0.129 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834
2025-12-08 04:19:55,661 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834
2025-12-08 04:19:55,661 - INFO - VM ctest-vn2_vm2-18290950 is removed in Compute, and routes are removed in all compute nodes
2025-12-08 04:19:55,871 - INFO - Routes for VM ctest-vn2_vm2-18290950 is removed in all control-nodes
2025-12-08 04:19:55,872 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:19:55,935 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:19:55,966 - DEBUG - VM a9733ad1-0267-4d1d-9de5-77b8687a88bd is still present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:19:55,966 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:19:55,966 - DEBUG - interface for vm a9733ad1-0267-4d1d-9de5-77b8687a88bd deleted from vrouter uve ...
2025-12-08 04:19:55,966 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:19:58,967 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:19:59,026 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:19:59,055 - DEBUG - VM a9733ad1-0267-4d1d-9de5-77b8687a88bd is still present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:19:59,055 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:19:59,055 - DEBUG - interface for vm a9733ad1-0267-4d1d-9de5-77b8687a88bd deleted from vrouter uve ...
2025-12-08 04:19:59,055 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:20:02,056 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:20:02,119 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:20:02,156 - DEBUG - VM a9733ad1-0267-4d1d-9de5-77b8687a88bd is still present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:20:02,156 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:20:02,157 - DEBUG - interface for vm a9733ad1-0267-4d1d-9de5-77b8687a88bd deleted from vrouter uve ...
2025-12-08 04:20:02,157 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:20:05,157 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-1:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:20:05,254 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-1?flat
2025-12-08 04:20:05,294 - DEBUG - VM a9733ad1-0267-4d1d-9de5-77b8687a88bd is not present in vrouter an-jenkins-deploy-platform-ansible-os-4854-1 uve
2025-12-08 04:20:05,294 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:20:05,295 - DEBUG - interface for vm a9733ad1-0267-4d1d-9de5-77b8687a88bd deleted from vrouter uve ...
2025-12-08 04:20:05,295 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:20:05,295 - DEBUG - Validated that VM a9733ad1-0267-4d1d-9de5-77b8687a88bd is removed in Vrouter UVE
2025-12-08 04:20:05,295 - DEBUG - Verifying through opserver in 10.0.0.23
2025-12-08 04:20:05,295 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834?flat
2025-12-08 04:20:05,332 - DEBUG - a9733ad1-0267-4d1d-9de5-77b8687a88bd BM not in default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn2-01263834 uve
2025-12-08 04:20:05,332 - DEBUG - Validated that VM a9733ad1-0267-4d1d-9de5-77b8687a88bd is not present in the VN UVE
2025-12-08 04:20:05,333 - DEBUG - Verifying the a9733ad1-0267-4d1d-9de5-77b8687a88bd virtual network link through opserver 10.0.0.23
2025-12-08 04:20:05,333 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines
2025-12-08 04:20:05,347 - INFO - a9733ad1-0267-4d1d-9de5-77b8687a88bd vm uve deleted from opserver
2025-12-08 04:20:05,347 - DEBUG - Verifying the a9733ad1-0267-4d1d-9de5-77b8687a88bd virtual network link through opserver 10.0.0.22
2025-12-08 04:20:05,347 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-12-08 04:20:05,361 - INFO - a9733ad1-0267-4d1d-9de5-77b8687a88bd vm uve deleted from opserver
2025-12-08 04:20:05,361 - DEBUG - Verifying the a9733ad1-0267-4d1d-9de5-77b8687a88bd virtual network link through opserver 10.0.0.65
2025-12-08 04:20:05,361 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machines
2025-12-08 04:20:05,376 - INFO - a9733ad1-0267-4d1d-9de5-77b8687a88bd vm uve deleted from opserver
2025-12-08 04:20:05,376 - INFO - Validated that VM a9733ad1-0267-4d1d-9de5-77b8687a88bd is removed from Opserver
2025-12-08 04:20:05,376 - INFO - Deleting VM ctest-vn1_vm2-50808408
2025-12-08 04:20:05,471 - DEBUG - Verifying in api server 10.0.0.65
2025-12-08 04:20:05,471 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128
2025-12-08 04:20:05,481 - DEBUG - VM ID b59971a0-b5d8-4726-a009-3db7c055d128 of VM ctest-vn1_vm2-50808408 is still found in API Server
2025-12-08 04:20:07,482 - DEBUG - Verifying in api server 10.0.0.65
2025-12-08 04:20:07,482 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128
2025-12-08 04:20:07,491 - DEBUG - Response Code: 404
2025-12-08 04:20:07,491 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128
2025-12-08 04:20:07,499 - DEBUG - Response Code: 404
2025-12-08 04:20:07,499 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-machine/b59971a0-b5d8-4726-a009-3db7c055d128
2025-12-08 04:20:07,506 - DEBUG - Response Code: 404
2025-12-08 04:20:07,506 - INFO - VM ctest-vn1_vm2-50808408 is fully removed in API-Server
2025-12-08 04:20:07,661 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found
2025-12-08 04:20:07,695 - DEBUG - Validated that vrouter 10.0.0.41 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066
2025-12-08 04:20:07,840 - DEBUG - VM route 192.168.1.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 of compute 10.0.0.129
2025-12-08 04:20:07,846 - DEBUG - Validated that vrouter 10.0.0.129 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066
2025-12-08 04:20:07,846 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066
2025-12-08 04:20:07,846 - INFO - VM ctest-vn1_vm2-50808408 is removed in Compute, and routes are removed in all compute nodes
2025-12-08 04:20:08,062 - INFO - Routes for VM ctest-vn1_vm2-50808408 is removed in all control-nodes
2025-12-08 04:20:08,063 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-2:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:20:08,124 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-2?flat
2025-12-08 04:20:08,156 - DEBUG - VM b59971a0-b5d8-4726-a009-3db7c055d128 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4854-2 uve
2025-12-08 04:20:08,156 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:20:08,156 - DEBUG - interface for vm b59971a0-b5d8-4726-a009-3db7c055d128 deleted from vrouter uve ...
2025-12-08 04:20:08,156 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:20:11,157 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-2:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:20:11,211 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-2?flat
2025-12-08 04:20:11,250 - DEBUG - VM b59971a0-b5d8-4726-a009-3db7c055d128 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4854-2 uve
2025-12-08 04:20:11,250 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:20:11,251 - DEBUG - interface for vm b59971a0-b5d8-4726-a009-3db7c055d128 deleted from vrouter uve ...
2025-12-08 04:20:11,251 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:20:14,252 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4854-2:Compute:contrail-vrouter-agent:0?flat
2025-12-08 04:20:14,306 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4854-2?flat
2025-12-08 04:20:14,407 - DEBUG - VM b59971a0-b5d8-4726-a009-3db7c055d128 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4854-2 uve
2025-12-08 04:20:14,407 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-12-08 04:20:14,407 - DEBUG - interface for vm b59971a0-b5d8-4726-a009-3db7c055d128 deleted from vrouter uve ...
2025-12-08 04:20:14,407 - DEBUG - Deleted interface not in error interface list ...
2025-12-08 04:20:14,407 - DEBUG - Validated that VM b59971a0-b5d8-4726-a009-3db7c055d128 is removed in Vrouter UVE
2025-12-08 04:20:14,407 - DEBUG - Verifying through opserver in 10.0.0.23
2025-12-08 04:20:14,407 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066?flat
2025-12-08 04:20:14,433 - DEBUG - b59971a0-b5d8-4726-a009-3db7c055d128 BM not in default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066 uve
2025-12-08 04:20:14,433 - DEBUG - Validated that VM b59971a0-b5d8-4726-a009-3db7c055d128 is not present in the VN UVE
2025-12-08 04:20:14,433 - DEBUG - Verifying the b59971a0-b5d8-4726-a009-3db7c055d128 virtual network link through opserver 10.0.0.23
2025-12-08 04:20:14,433 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-machines
2025-12-08 04:20:14,446 - INFO - b59971a0-b5d8-4726-a009-3db7c055d128 vm uve deleted from opserver
2025-12-08 04:20:14,446 - DEBUG - Verifying the b59971a0-b5d8-4726-a009-3db7c055d128 virtual network link through opserver 10.0.0.22
2025-12-08 04:20:14,446 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-12-08 04:20:14,459 - INFO - b59971a0-b5d8-4726-a009-3db7c055d128 vm uve deleted from opserver
2025-12-08 04:20:14,459 - DEBUG - Verifying the b59971a0-b5d8-4726-a009-3db7c055d128 virtual network link through opserver 10.0.0.65
2025-12-08 04:20:14,459 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-machines
2025-12-08 04:20:14,471 - INFO - b59971a0-b5d8-4726-a009-3db7c055d128 vm uve deleted from opserver
2025-12-08 04:20:14,472 - INFO - Validated that VM b59971a0-b5d8-4726-a009-3db7c055d128 is removed from Opserver
2025-12-08 04:20:14,472 - INFO - Deleting VN ctest-vn3-32268234
2025-12-08 04:20:14,590 - INFO - Deleting VM ctest-vn2_vm1-91232456
2025-12-08 04:20:14,710 - INFO - Deleting VM ctest-vn1_vm1-42592862
2025-12-08 04:20:14,953 - INFO - Deleting VN ctest-vn2-01263834
2025-12-08 04:20:15,013 - DEBUG - RefsExistError Delete when resource still referred: ['http://10.0.0.65:8082/virtual-machine-interface/37e4768a-4089-4b09-a908-a40b64820359'] while deleting VN ctest-vn2-01263834..Will retry
2025-12-08 04:20:17,141 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/8454ee1b-3fd7-41ef-8a6d-c483d648abff
2025-12-08 04:20:17,147 - DEBUG - Response Code: 404
2025-12-08 04:20:17,148 - DEBUG - Requesting: http://10.0.0.23:8082/domains
2025-12-08 04:20:17,155 - DEBUG - Requesting: http://10.0.0.23:8082/domain/c5befdc1-279e-4ac1-9620-9608231491f8
2025-12-08 04:20:17,169 - DEBUG - Requesting: http://10.0.0.23:8082/project/39b9363a-5400-4ea1-a180-7dd8e2fe11b9
2025-12-08 04:20:17,274 - INFO - Validated that VN ctest-vn2-01263834 is not found in API Server
2025-12-08 04:20:17,284 - DEBUG - VN ctest-vn2-01263834 is not present in Agent 10.0.0.41
2025-12-08 04:20:17,295 - DEBUG - VN ctest-vn2-01263834 is not present in Agent 10.0.0.129
2025-12-08 04:20:17,295 - INFO - Validated that VN ctest-vn2-01263834 is not in any agent
2025-12-08 04:20:17,327 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-01263834 info
2025-12-08 04:20:17,463 - INFO - Deleting VN ctest-vn1-11209066
2025-12-08 04:20:17,597 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/904d4940-9135-42bf-812f-06d264925cbe
2025-12-08 04:20:17,608 - DEBUG - Response Code: 404
2025-12-08 04:20:17,608 - DEBUG - Requesting: http://10.0.0.23:8082/domains
2025-12-08 04:20:17,615 - DEBUG - Requesting: http://10.0.0.23:8082/domain/c5befdc1-279e-4ac1-9620-9608231491f8
2025-12-08 04:20:17,634 - DEBUG - Requesting: http://10.0.0.23:8082/project/39b9363a-5400-4ea1-a180-7dd8e2fe11b9
2025-12-08 04:20:17,735 - INFO - Validated that VN ctest-vn1-11209066 is not found in API Server
2025-12-08 04:20:17,747 - DEBUG - VN ctest-vn1-11209066 is not present in Agent 10.0.0.41
2025-12-08 04:20:17,759 - DEBUG - VN ctest-vn1-11209066 is not present in Agent 10.0.0.129
2025-12-08 04:20:17,759 - INFO - Validated that VN ctest-vn1-11209066 is not in any agent
2025-12-08 04:20:17,766 - DEBUG - VRF 2 is not seen in agent 10.0.0.129
2025-12-08 04:20:17,772 - DEBUG - Vrouter 10.0.0.129 does not have vrf 2 for VN ctest-vn1-11209066
2025-12-08 04:20:17,772 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestBasicVMVN0-32026406:ctest-vn1-11209066
2025-12-08 04:20:17,794 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-11209066 info
2025-12-08 04:20:17,839 - INFO - Deleted policy policy2
2025-12-08 04:20:17,887 - INFO - Deleted policy policy1
2025-12-08 04:20:19,156 - DEBUG - Skipping xmpp flap check
2025-12-08 04:20:19,156 - INFO - END TEST : test_process_restart_in_policy_between_vns : PASSED[0:05:57]
2025-12-08 04:20:19,156 - INFO - --------------------------------------------------------------------------------
2025-12-08 04:20:19,158 - INFO - ================================================================================
2025-12-08 04:20:19,158 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling
2025-12-08 04:20:19,158 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229).
1. Send broadcast traffic from one compute node.
2. Other compute in same subnet should receive that traffic.
3. Receiving compute should treat this traffic as underlay.
4. Compute should not replicate the packet and send the copy back.
Pass criteria: Step 3-4 should pass
Maintainer : chhandak@juniper.net
2025-12-08 04:20:20,442 - DEBUG - Skipping xmpp flap check
2025-12-08 04:20:20,442 - INFO - Initial checks done. Running the testcase now
2025-12-08 04:20:20,442 - INFO -
2025-12-08 04:20:20,840 - INFO - Agent interface name: ens6
2025-12-08 04:20:20,918 - DEBUG - Executing command: sudo tcpdump -nni ens6 -U host 10.20.0.255 -w /tmp/ens6_ctest-random-64908468.pcap
2025-12-08 04:20:26,303 - INFO - Pinging broacast address 10.20.0.255 from compute 10.20.0.252
2025-12-08 04:20:40,614 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}')
2025-12-08 04:20:42,754 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-64908468.pcap | grep -c length
2025-12-08 04:20:42,768 - DEBUG - STDOUT: 10
2025-12-08 04:20:42,768 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-64908468.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2025-12-08 04:20:42,768 - INFO - 10 packets are found in tcpdump output as expected
2025-12-08 04:20:42,768 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}')
2025-12-08 04:20:44,814 - INFO - Packet count matched: Compute 10.0.0.129 has receive only 10 packet from source IP 10.20.0.255. No duplicate packet seen
2025-12-08 04:20:46,072 - DEBUG - Skipping xmpp flap check
2025-12-08 04:20:46,072 - INFO - END TEST : test_underlay_broadcast_traffic_handling : PASSED[0:00:27]
2025-12-08 04:20:46,073 - INFO - --------------------------------------------------------------------------------
2025-12-08 04:20:46,961 - INFO - Deleted project: ctest-TestBasicVMVN0-32026406, ID : 39b9363a-5400-4ea1-a180-7dd8e2fe11b9