2025-09-18 04:07:10,970 - INFO - Domain Default found not creating
2025-09-18 04:07:11,174 - INFO - Project ctest-TestBasicVMVN0-22347485 not found, creating it
2025-09-18 04:07:11,786 - INFO - Created Project:ctest-TestBasicVMVN0-22347485, ID : 0a1e7454-93bc-4746-b98c-6e3a6282a39d
2025-09-18 04:07:13,885 - INFO - ================================================================================
2025-09-18 04:07:13,886 - INFO - STARTING TEST : test_control_node_switchover
2025-09-18 04:07:13,886 - INFO - TEST DESCRIPTION : Stop the control node and check peering with agent fallback to other control node.
1. Pick one VN from respource pool which has 2 VM's in it
2. Verify ping between VM's
3. Find active control node in cluster by agent inspect
4. Stop control service on active control node
5. Verify agents are connected to new active control-node using xmpp connections
6. Bring back control service on previous active node
7. Verify ping between VM's again after bringing up control serveice
Pass criteria: Step 2,5 and 7 should pass
2025-09-18 04:07:15,176 - DEBUG - Skipping xmpp flap check
2025-09-18 04:07:15,176 - INFO - Initial checks done. Running the testcase now
2025-09-18 04:07:15,176 - INFO -
2025-09-18 04:07:15,907 - DEBUG - Response for create_network : {'network': {'id': '3a2178fe-5c50-4c6e-8b9a-d91af4303325', 'name': 'ctest-vn1-28807685', 'tenant_id': '0a1e745493bc4746b98c6e3a6282a39d', 'project_id': '0a1e745493bc4746b98c6e3a6282a39d', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestBasicVMVN0-22347485', 'ctest-vn1-28807685'], 'port_security_enabled': True, 'description': ''}}
2025-09-18 04:07:16,271 - DEBUG - Response for create_subnet : {'subnet': {'id': 'be16e49d-2880-4331-b72a-413e95770b5c', 'name': '', 'tenant_id': '0a1e745493bc4746b98c6e3a6282a39d', 'network_id': '3a2178fe-5c50-4c6e-8b9a-d91af4303325', 'ip_version': 4, 'cidr': '192.168.1.0/24', 'allocation_pools': [{'start': '192.168.1.2', 'end': '192.168.1.254'}], 'gateway_ip': '192.168.1.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '192.168.1.2', 'tags': [], 'project_id': '0a1e745493bc4746b98c6e3a6282a39d'}}
2025-09-18 04:07:16,299 - INFO - Created VN ctest-vn1-28807685
2025-09-18 04:07:16,318 - DEBUG - VN ctest-vn1-28807685 UUID is 3a2178fe-5c50-4c6e-8b9a-d91af4303325
2025-09-18 04:07:16,415 - DEBUG - Requesting: http://10.0.0.56:8082/domains
2025-09-18 04:07:16,787 - DEBUG - Requesting: http://10.0.0.56:8082/domains
2025-09-18 04:07:16,826 - DEBUG - Requesting: http://10.0.0.56:8082/domain/245899c5-19ba-4930-be6a-859f116860e1
2025-09-18 04:07:16,844 - DEBUG - Requesting: http://10.0.0.56:8082/project/0a1e7454-93bc-4746-b98c-6e3a6282a39d
2025-09-18 04:07:16,944 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/3a2178fe-5c50-4c6e-8b9a-d91af4303325
2025-09-18 04:07:16,964 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/3a2178fe-5c50-4c6e-8b9a-d91af4303325
2025-09-18 04:07:16,981 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/652b3292-3160-49c9-8985-481dd28e8beb
2025-09-18 04:07:16,991 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/652b3292-3160-49c9-8985-481dd28e8beb
2025-09-18 04:07:17,004 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/9aa0ecda-8421-49b4-a1f5-a8cefa1b7c48
2025-09-18 04:07:17,014 - DEBUG - Route Targets: ['target:64512:8000004']
2025-09-18 04:07:17,014 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/3a2178fe-5c50-4c6e-8b9a-d91af4303325
2025-09-18 04:07:17,030 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/652b3292-3160-49c9-8985-481dd28e8beb
2025-09-18 04:07:17,053 - INFO - Verified VN network id 12 for VN 3a2178fe-5c50-4c6e-8b9a-d91af4303325
2025-09-18 04:07:17,054 - INFO - Verifications in API Server for VN ctest-vn1-28807685 passed
2025-09-18 04:07:17,055 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/3a2178fe-5c50-4c6e-8b9a-d91af4303325
2025-09-18 04:07:17,075 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/652b3292-3160-49c9-8985-481dd28e8beb
2025-09-18 04:07:17,091 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/652b3292-3160-49c9-8985-481dd28e8beb
2025-09-18 04:07:17,101 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/9aa0ecda-8421-49b4-a1f5-a8cefa1b7c48
2025-09-18 04:07:17,124 - DEBUG - Control-node 10.0.0.56 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-22347485', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4188762162297064558', 'uuid-lslong': '10059591427474600741'}, 'enable': 'true', 'created': '2025-09-18T04:07:15', 'last-modified': '2025-09-18T04:07:16', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.951748'}
2025-09-18 04:07:17,135 - DEBUG - Route Targets: ['target:64512:8000004']
2025-09-18 04:07:17,145 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-22347485', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4188762162297064558', 'uuid-lslong': '10059591427474600741'}, 'enable': 'true', 'created': '2025-09-18T04:07:15', 'last-modified': '2025-09-18T04:07:16', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.982212'}
2025-09-18 04:07:17,153 - DEBUG - Route Targets: ['target:64512:8000004']
2025-09-18 04:07:17,160 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-22347485', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '4188762162297064558', 'uuid-lslong': '10059591427474600741'}, 'enable': 'true', 'created': '2025-09-18T04:07:15', 'last-modified': '2025-09-18T04:07:16', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.996204'}
2025-09-18 04:07:17,166 - DEBUG - Route Targets: ['target:64512:8000004']
2025-09-18 04:07:17,166 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-28807685 passed
2025-09-18 04:07:17,167 - DEBUG - ====Verifying policy data for ctest-vn1-28807685 in API_Server ======
2025-09-18 04:07:17,167 - DEBUG - Requesting: http://10.0.0.56:8082/domains
2025-09-18 04:07:17,177 - DEBUG - Requesting: http://10.0.0.56:8082/domain/245899c5-19ba-4930-be6a-859f116860e1
2025-09-18 04:07:17,194 - DEBUG - Requesting: http://10.0.0.56:8082/project/0a1e7454-93bc-4746-b98c-6e3a6282a39d
2025-09-18 04:07:17,301 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/3a2178fe-5c50-4c6e-8b9a-d91af4303325
2025-09-18 04:07:17,317 - DEBUG - =>VN ctest-vn1-28807685 has no policy to be verified
2025-09-18 04:07:17,317 - DEBUG - Verifying the vn in opserver
2025-09-18 04:07:17,317 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 virtual network link through opserver 10.0.0.56
2025-09-18 04:07:17,318 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-networks
2025-09-18 04:07:17,675 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-networks
2025-09-18 04:07:17,749 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685?flat'}
2025-09-18 04:07:17,749 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 virtual network link through opserver 10.0.0.22
2025-09-18 04:07:17,749 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks
2025-09-18 04:07:18,101 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks
2025-09-18 04:07:18,166 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685?flat'}
2025-09-18 04:07:18,166 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 virtual network link through opserver 10.0.0.41
2025-09-18 04:07:18,166 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks
2025-09-18 04:07:18,520 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks
2025-09-18 04:07:18,588 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685?flat'}
2025-09-18 04:07:18,588 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is found in opserver
2025-09-18 04:07:18,612 - DEBUG - Do not have enough data to verify VN in agent
2025-09-18 04:07:18,624 - DEBUG - VRF ids for VN ctest-vn1-28807685: {}
2025-09-18 04:07:18,687 - DEBUG - Services list from nova: [, , , ]
2025-09-18 04:07:19,892 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4130-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4130-1)
2025-09-18 04:07:21,489 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4130-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4130-2)
2025-09-18 04:07:21,489 - INFO - Waiting for VM ctest-vn1_vm1-56080873 to be up..
2025-09-18 04:07:21,608 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2025-09-18 04:07:26,697 - DEBUG - VM is in ACTIVE state now
2025-09-18 04:07:26,697 - INFO - VM name : ctest-vn1_vm1-56080873
2025-09-18 04:07:26,793 - DEBUG - VM ctest-vn1_vm1-56080873 ID is 4969978e-36f9-4abc-8e76-93e225439316
2025-09-18 04:07:26,822 - DEBUG - VM ctest-vn1_vm1-56080873 launched on Node an-jenkins-deploy-platform-ansible-os-4130-1
2025-09-18 04:07:26,945 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316
2025-09-18 04:07:27,294 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316
2025-09-18 04:07:27,336 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/13ec27ea-d68b-4749-b915-ea3e6c732c59
2025-09-18 04:07:30,512 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms')
2025-09-18 04:07:30,512 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-56080873 failed!
2025-09-18 04:07:30,528 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is 192.168.1.1 and allocation pool is NOT set
2025-09-18 04:07:34,606 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms')
2025-09-18 04:07:34,606 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-56080873 failed!
2025-09-18 04:07:34,622 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is 192.168.1.1 and allocation pool is NOT set
2025-09-18 04:07:38,695 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms')
2025-09-18 04:07:38,695 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-56080873 failed!
2025-09-18 04:07:38,711 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is 192.168.1.1 and allocation pool is NOT set
2025-09-18 04:07:42,792 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1022ms')
2025-09-18 04:07:42,792 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-56080873 failed!
2025-09-18 04:07:42,808 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is 192.168.1.1 and allocation pool is NOT set
2025-09-18 04:07:46,894 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms')
2025-09-18 04:07:46,894 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-56080873 failed!
2025-09-18 04:07:46,912 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is 192.168.1.1 and allocation pool is NOT set
2025-09-18 04:07:50,990 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms')
2025-09-18 04:07:50,990 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-56080873 failed!
2025-09-18 04:07:51,006 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is 192.168.1.1 and allocation pool is NOT set
2025-09-18 04:07:55,089 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms')
2025-09-18 04:07:55,089 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-56080873 failed!
2025-09-18 04:07:55,104 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is 192.168.1.1 and allocation pool is NOT set
2025-09-18 04:07:59,183 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms')
2025-09-18 04:07:59,183 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-56080873 failed!
2025-09-18 04:07:59,198 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is 192.168.1.1 and allocation pool is NOT set
2025-09-18 04:08:03,277 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms')
2025-09-18 04:08:03,277 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-56080873 failed!
2025-09-18 04:08:03,293 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is 192.168.1.1 and allocation pool is NOT set
2025-09-18 04:08:05,365 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=8.29 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=3.17 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 3.172/5.731/8.291/2.559 ms')
2025-09-18 04:08:05,365 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-56080873 passed
2025-09-18 04:08:05,439 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:08:05,439 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-56080873, IP 192.168.1.3, Port 22
2025-09-18 04:08:05,508 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-09-18 04:08:05,602 - DEBUG - VM ctest-vn1_vm1-56080873 is NOT ready for SSH connections, VM status: ACTIVE
2025-09-18 04:08:10,603 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:08:10,603 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-56080873, IP 192.168.1.3, Port 22
2025-09-18 04:08:10,671 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-09-18 04:08:10,760 - DEBUG - VM ctest-vn1_vm1-56080873 is NOT ready for SSH connections, VM status: ACTIVE
2025-09-18 04:08:15,761 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:08:15,761 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-56080873, IP 192.168.1.3, Port 22
2025-09-18 04:08:15,828 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-09-18 04:08:15,921 - DEBUG - VM ctest-vn1_vm1-56080873 is NOT ready for SSH connections, VM status: ACTIVE
2025-09-18 04:08:20,921 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:08:20,922 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-56080873, IP 192.168.1.3, Port 22
2025-09-18 04:08:20,991 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-09-18 04:08:21,094 - DEBUG - VM ctest-vn1_vm1-56080873 is NOT ready for SSH connections, VM status: ACTIVE
2025-09-18 04:08:26,095 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:08:26,095 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-56080873, IP 192.168.1.3, Port 22
2025-09-18 04:08:26,263 - DEBUG - VM ctest-vn1_vm1-56080873 is ready for SSH connections
2025-09-18 04:08:26,263 - INFO - Waiting for VM ctest-vn1_vm2-74398981 to be up..
2025-09-18 04:08:26,376 - DEBUG - VM is in ACTIVE state now
2025-09-18 04:08:26,376 - INFO - VM name : ctest-vn1_vm2-74398981
2025-09-18 04:08:26,475 - DEBUG - VM ctest-vn1_vm2-74398981 ID is 7394bd4a-d6be-49f9-b845-c47c148de115
2025-09-18 04:08:26,475 - DEBUG - VM ctest-vn1_vm2-74398981 launched on Node an-jenkins-deploy-platform-ansible-os-4130-2
2025-09-18 04:08:26,573 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/7394bd4a-d6be-49f9-b845-c47c148de115
2025-09-18 04:08:26,587 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/eacdf482-1a52-439d-b69d-7164b3ca6e53
2025-09-18 04:08:27,772 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=3.49 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.572 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.572/2.030/3.489/1.458 ms')
2025-09-18 04:08:27,772 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm2-74398981 passed
2025-09-18 04:08:27,842 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:08:27,845 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-74398981, IP 192.168.1.4, Port 22
2025-09-18 04:08:27,915 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-09-18 04:08:28,019 - DEBUG - VM ctest-vn1_vm2-74398981 is NOT ready for SSH connections, VM status: ACTIVE
2025-09-18 04:08:33,019 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:08:33,019 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-74398981, IP 192.168.1.4, Port 22
2025-09-18 04:08:33,187 - DEBUG - VM ctest-vn1_vm2-74398981 is ready for SSH connections
2025-09-18 04:08:33,187 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:08:33,187 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.61, gateway password: c0ntrail123
2025-09-18 04:08:33,187 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4
2025-09-18 04:08:51,873 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=10.9 ms
64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=2.71 ms
64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=2.96 ms
--- 192.168.1.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 2.713/5.527/10.903/3.803 ms
2025-09-18 04:08:51,873 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-56080873 passed
2025-09-18 04:08:51,873 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:08:51,873 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-09-18 04:08:51,873 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3
2025-09-18 04:09:31,611 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=6.89 ms
64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.46 ms
64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.17 ms
--- 192.168.1.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.177/3.177/6.894/2.631 ms
2025-09-18 04:09:31,611 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-74398981 passed
2025-09-18 04:09:31,621 - INFO - Active control node from the Agent 10.0.0.61 is 10.0.0.22
2025-09-18 04:09:31,621 - INFO - Stoping the Control service in 10.0.0.22
2025-09-18 04:09:32,024 - INFO - 10.0.0.22
2025-09-18 04:09:32,025 - INFO - control:inactive
2025-09-18 04:09:32,025 - INFO - Contrail services control are down on nodes 10.0.0.22
2025-09-18 04:09:37,033 - INFO - Active control node from the Agent 10.0.0.61 is 10.0.0.41
2025-09-18 04:09:37,033 - INFO - Starting the Control service in 10.0.0.22
2025-09-18 04:09:37,379 - INFO - 10.0.0.22
2025-09-18 04:09:37,383 - INFO - control:initializing
2025-09-18 04:09:37,383 - DEBUG - defaultdict(, {'10.0.0.22': {'control': {'status': 'initializing', 'description': None}}})
2025-09-18 04:09:37,384 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-09-18 04:09:42,484 - INFO - 10.0.0.22
2025-09-18 04:09:42,492 - INFO - control:active
2025-09-18 04:09:42,493 - INFO - Contrail services ['control'] are up on nodes 10.0.0.22
2025-09-18 04:09:47,610 - DEBUG - VM is in ACTIVE state now
2025-09-18 04:09:47,610 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:09:47,610 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316
2025-09-18 04:09:47,619 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/13ec27ea-d68b-4749-b915-ea3e6c732c59
2025-09-18 04:09:47,630 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316
2025-09-18 04:09:47,638 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/13ec27ea-d68b-4749-b915-ea3e6c732c59
2025-09-18 04:09:47,648 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/2cd35f94-a58e-40c7-b2e1-c6a7da4ae3e2
2025-09-18 04:09:47,659 - DEBUG - Verifying in api server 10.0.0.41
2025-09-18 04:09:47,659 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316
2025-09-18 04:09:47,667 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/13ec27ea-d68b-4749-b915-ea3e6c732c59
2025-09-18 04:09:47,679 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/2cd35f94-a58e-40c7-b2e1-c6a7da4ae3e2
2025-09-18 04:09:47,694 - INFO - VM ctest-vn1_vm1-56080873 verfication in all API Servers passed
2025-09-18 04:09:47,714 - DEBUG - VM ctest-vn1_vm1-56080873 Tap interface: {'index': '3', 'name': 'tap13ec27ea-d6', 'uuid': '13ec27ea-d68b-4749-b915-ea3e6c732c59', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685:ctest-vn1-28807685', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685', 'vm_uuid': '4969978e-36f9-4abc-8e76-93e225439316', 'vm_name': 'ctest-vn1_vm1-56080873', 'ip_addr': '192.168.1.3', 'mac_addr': '02:13:ec:27:ea:d6', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '163', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-22347485:13ec27ea-d68b-4749-b915-ea3e6c732c59', 'sg_uuid_list': ['f51220a7-8281-4ea3-8440-e34dbd8dc48d'], 'static_route_list': None, 'vm_project_uuid': '0a1e7454-93bc-4746-b98c-6e3a6282a39d', 'admin_state': 'Enabled', 'flow_key_idx': '30', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-09-18 04:09:47,715 - DEBUG - Agent 10.0.0.61 vrf name: default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685:ctest-vn1-28807685
2025-09-18 04:09:47,720 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685:ctest-vn1-28807685', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685', 'table_label': '-1', 'vxlan_id': '12', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.129:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]}
2025-09-18 04:09:47,803 - DEBUG - VM ctest-vn1_vm1-56080873 : Tap interface tap13ec27ea-d6 is set to Active
2025-09-18 04:09:47,804 - DEBUG - Tap interface tap13ec27ea-d6 detail : {'index': '3', 'name': 'tap13ec27ea-d6', 'uuid': '13ec27ea-d68b-4749-b915-ea3e6c732c59', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685:ctest-vn1-28807685', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685', 'vm_uuid': '4969978e-36f9-4abc-8e76-93e225439316', 'vm_name': 'ctest-vn1_vm1-56080873', 'ip_addr': '192.168.1.3', 'mac_addr': '02:13:ec:27:ea:d6', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '163', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-22347485:13ec27ea-d68b-4749-b915-ea3e6c732c59', 'sg_uuid_list': ['f51220a7-8281-4ea3-8440-e34dbd8dc48d'], 'static_route_list': None, 'vm_project_uuid': '0a1e7454-93bc-4746-b98c-6e3a6282a39d', 'admin_state': 'Enabled', 'flow_key_idx': '30', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-09-18 04:09:47,926 - DEBUG - Starting Layer 2 verification in Agent
2025-09-18 04:09:47,938 - DEBUG - Layer 2 path is seen for VM MAC 02:13:ec:27:ea:d6 in agent 10.0.0.61
2025-09-18 04:09:47,939 - DEBUG - Active layer 2 route in agent is present for VMI tap13ec27ea-d6
2025-09-18 04:09:47,939 - DEBUG - L2 label(29) matches bw route table and interface table
2025-09-18 04:09:47,947 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is found in Agent of node 10.0.0.61
2025-09-18 04:09:48,087 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is consistent in agent 10.0.0.61
2025-09-18 04:09:48,087 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.61
2025-09-18 04:09:48,087 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 verification for VM ctest-vn1_vm1-56080873 in Agent 10.0.0.61 passed
2025-09-18 04:09:48,198 - DEBUG - Starting all layer 2 verification in agent 10.0.0.61
2025-09-18 04:09:48,210 - DEBUG - Route for VM MAC 02:13:ec:27:ea:d6 is consistent in agent 10.0.0.61
2025-09-18 04:09:48,218 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is found in Agent of node 10.0.0.27
2025-09-18 04:09:48,354 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is consistent in agent 10.0.0.27
2025-09-18 04:09:48,355 - DEBUG - Route for VM IP ['192.168.1.3'] is consistent in agent 10.0.0.27
2025-09-18 04:09:48,355 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 verification for VM ctest-vn1_vm1-56080873 in Agent 10.0.0.27 passed
2025-09-18 04:09:48,467 - DEBUG - Starting all layer 2 verification in agent 10.0.0.27
2025-09-18 04:09:48,479 - DEBUG - Route for VM MAC 02:13:ec:27:ea:d6 is consistent in agent 10.0.0.27
2025-09-18 04:09:48,479 - INFO - VM ctest-vn1_vm1-56080873 verifications in Compute nodes passed
2025-09-18 04:09:48,720 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.61
2025-09-18 04:09:48,726 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0
2025-09-18 04:09:48,861 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.27
2025-09-18 04:09:48,868 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.129, 24; Got 10.20.0.129,24
2025-09-18 04:09:48,868 - DEBUG - Validated VM route 192.168.1.3 in vrouter of 10.0.0.27
2025-09-18 04:09:48,868 - INFO - Validated routes of VM ctest-vn1_vm1-56080873 in all vrouters
2025-09-18 04:09:49,380 - DEBUG - Starting all layer2 verification in 10.0.0.22 Control Node
2025-09-18 04:09:49,519 - DEBUG - Layer2 route found for VM MAC 02:13:ec:27:ea:d6 in Control-node 10.0.0.22
2025-09-18 04:09:49,519 - DEBUG - L2 Label for VM ctest-vn1_vm1-56080873 same between Control-node 10.0.0.22 and Agent, Expected: 29, Seen: 29
2025-09-18 04:09:49,519 - DEBUG - Starting all layer2 verification in 10.0.0.41 Control Node
2025-09-18 04:09:49,649 - DEBUG - Layer2 route found for VM MAC 02:13:ec:27:ea:d6 in Control-node 10.0.0.41
2025-09-18 04:09:49,649 - DEBUG - L2 Label for VM ctest-vn1_vm1-56080873 same between Control-node 10.0.0.41 and Agent, Expected: 29, Seen: 29
2025-09-18 04:09:49,649 - INFO - Verification in Control-nodes for VM ctest-vn1_vm1-56080873 passed
2025-09-18 04:09:49,649 - DEBUG - Verifying the vm in opserver
2025-09-18 04:09:49,650 - DEBUG - Verifying in collector 10.0.0.41 ...
2025-09-18 04:09:49,650 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316?flat
2025-09-18 04:09:49,722 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-22347485:13ec27ea-d68b-4749-b915-ea3e6c732c59?flat
2025-09-18 04:09:49,796 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-22347485:13ec27ea-d68b-4749-b915-ea3e6c732c59?flat
2025-09-18 04:09:49,819 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316?flat
2025-09-18 04:09:49,838 - DEBUG - Verifying vm in vn uve
2025-09-18 04:09:49,838 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-22347485:13ec27ea-d68b-4749-b915-ea3e6c732c59?flat
2025-09-18 04:09:49,911 - DEBUG - VM uve shows interface as {'vm_name': 'ctest-vn1_vm1-56080873', 'is_health_check_active': True, 'ip6_address': '::', 'l2_active': True, 'rx_vlan': 65535, 'ip_address': '192.168.1.3', 'vm_uuid': '4969978e-36f9-4abc-8e76-93e225439316', 'active': True, 'fixed_ip4_list': ['192.168.1.3'], 'vhostuser_mode': 0, 'tx_vlan': 65535, 'admin_state': True, 'ip6_active': False, 'gateway': '192.168.1.1', 'uuid': '13ec27ea-d68b-4749-b915-ea3e6c732c59', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685', '__T': 1758168572737588, 'label': 24, 'ip4_active': True, 'mac_address': '02:13:ec:27:ea:d6', 'vn_uuid': '3a2178fe-5c50-4c6e-8b9a-d91af4303325', 'port_mirror_enabled': False}
2025-09-18 04:09:49,911 - DEBUG - VM uve shows ip address as ['192.168.1.3', '::']
2025-09-18 04:09:49,911 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685
2025-09-18 04:09:49,911 - INFO - Verifying through opserver in 10.0.0.56
2025-09-18 04:09:49,911 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685?flat
2025-09-18 04:09:49,942 - DEBUG - expected vm list 4969978e-36f9-4abc-8e76-93e225439316
2025-09-18 04:09:49,942 - DEBUG - Extracted vm list ['4969978e-36f9-4abc-8e76-93e225439316']
2025-09-18 04:09:49,942 - INFO - VM 4969978e-36f9-4abc-8e76-93e225439316 is present in default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685
2025-09-18 04:09:49,942 - DEBUG - Verifying vm in vrouter uve
2025-09-18 04:09:49,942 - DEBUG - Getting info from collector 10.0.0.41..
2025-09-18 04:09:49,942 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316?flat
2025-09-18 04:09:50,010 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4130-1']
2025-09-18 04:09:50,010 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:09:50,067 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:09:50,103 - DEBUG - VM 4969978e-36f9-4abc-8e76-93e225439316 is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:09:50,103 - DEBUG - Validated that VM ctest-vn1_vm1-56080873 is in Vrouter an-jenkins-deploy-platform-ansible-os-4130-1 UVE
2025-09-18 04:09:50,103 - DEBUG - Verifying vm tap interface/vn in vrouter uve
2025-09-18 04:09:50,111 - DEBUG - Expected tap interface of VM uuid 4969978e-36f9-4abc-8e76-93e225439316 is default-domain:ctest-TestBasicVMVN0-22347485:13ec27ea-d68b-4749-b915-ea3e6c732c59
2025-09-18 04:09:50,111 - DEBUG - Expected VN of VM uuid 4969978e-36f9-4abc-8e76-93e225439316 is default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685
2025-09-18 04:09:50,112 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:09:50,163 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:09:50,199 - DEBUG - VM 4969978e-36f9-4abc-8e76-93e225439316 is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:09:50,199 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-22347485:13ec27ea-d68b-4749-b915-ea3e6c732c59 of vm 4969978e-36f9-4abc-8e76-93e225439316 is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:09:50,199 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 of vm 4969978e-36f9-4abc-8e76-93e225439316 is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:09:50,199 - DEBUG - Verifying the 4969978e-36f9-4abc-8e76-93e225439316 virtual network link through opserver 10.0.0.56
2025-09-18 04:09:50,199 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines
2025-09-18 04:09:50,212 - DEBUG - VM link and name as {'name': '4969978e-36f9-4abc-8e76-93e225439316', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316?flat'}
2025-09-18 04:09:50,212 - DEBUG - Verifying the 4969978e-36f9-4abc-8e76-93e225439316 virtual network link through opserver 10.0.0.22
2025-09-18 04:09:50,212 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-09-18 04:09:50,228 - DEBUG - VM link and name as {'name': '4969978e-36f9-4abc-8e76-93e225439316', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316?flat'}
2025-09-18 04:09:50,230 - DEBUG - Verifying the 4969978e-36f9-4abc-8e76-93e225439316 virtual network link through opserver 10.0.0.41
2025-09-18 04:09:50,230 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines
2025-09-18 04:09:50,243 - DEBUG - VM link and name as {'name': '4969978e-36f9-4abc-8e76-93e225439316', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316?flat'}
2025-09-18 04:09:50,243 - INFO - VM ctest-vn1_vm1-56080873 validations in Opserver passed
2025-09-18 04:09:50,336 - DEBUG - VM is in ACTIVE state now
2025-09-18 04:09:50,336 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:09:50,336 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/7394bd4a-d6be-49f9-b845-c47c148de115
2025-09-18 04:09:50,345 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/eacdf482-1a52-439d-b69d-7164b3ca6e53
2025-09-18 04:09:50,355 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/7394bd4a-d6be-49f9-b845-c47c148de115
2025-09-18 04:09:50,362 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/eacdf482-1a52-439d-b69d-7164b3ca6e53
2025-09-18 04:09:50,373 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/cd81416d-0673-41de-b175-d24780d87d72
2025-09-18 04:09:50,385 - DEBUG - Verifying in api server 10.0.0.41
2025-09-18 04:09:50,385 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/7394bd4a-d6be-49f9-b845-c47c148de115
2025-09-18 04:09:50,396 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/eacdf482-1a52-439d-b69d-7164b3ca6e53
2025-09-18 04:09:50,409 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/cd81416d-0673-41de-b175-d24780d87d72
2025-09-18 04:09:50,418 - INFO - VM ctest-vn1_vm2-74398981 verfication in all API Servers passed
2025-09-18 04:09:50,439 - DEBUG - VM ctest-vn1_vm2-74398981 Tap interface: {'index': '3', 'name': 'tapeacdf482-1a', 'uuid': 'eacdf482-1a52-439d-b69d-7164b3ca6e53', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685:ctest-vn1-28807685', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685', 'vm_uuid': '7394bd4a-d6be-49f9-b845-c47c148de115', 'vm_name': 'ctest-vn1_vm2-74398981', 'ip_addr': '192.168.1.4', 'mac_addr': '02:ea:cd:f4:82:1a', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '125', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-22347485:eacdf482-1a52-439d-b69d-7164b3ca6e53', 'sg_uuid_list': ['f51220a7-8281-4ea3-8440-e34dbd8dc48d'], 'static_route_list': None, 'vm_project_uuid': '0a1e7454-93bc-4746-b98c-6e3a6282a39d', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-09-18 04:09:50,439 - DEBUG - Agent 10.0.0.27 vrf name: default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685:ctest-vn1-28807685
2025-09-18 04:09:50,445 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685:ctest-vn1-28807685', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685', 'table_label': '-1', 'vxlan_id': '12', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.192:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]}
2025-09-18 04:09:50,530 - DEBUG - VM ctest-vn1_vm2-74398981 : Tap interface tapeacdf482-1a is set to Active
2025-09-18 04:09:50,530 - DEBUG - Tap interface tapeacdf482-1a detail : {'index': '3', 'name': 'tapeacdf482-1a', 'uuid': 'eacdf482-1a52-439d-b69d-7164b3ca6e53', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685:ctest-vn1-28807685', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '24', 'l2_label': '29', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685', 'vm_uuid': '7394bd4a-d6be-49f9-b845-c47c148de115', 'vm_name': 'ctest-vn1_vm2-74398981', 'ip_addr': '192.168.1.4', 'mac_addr': '02:ea:cd:f4:82:1a', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.3', 'service_vlan_list': None, 'os_ifindex': '125', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-22347485:eacdf482-1a52-439d-b69d-7164b3ca6e53', 'sg_uuid_list': ['f51220a7-8281-4ea3-8440-e34dbd8dc48d'], 'static_route_list': None, 'vm_project_uuid': '0a1e7454-93bc-4746-b98c-6e3a6282a39d', 'admin_state': 'Enabled', 'flow_key_idx': '26', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-09-18 04:09:50,643 - DEBUG - Starting Layer 2 verification in Agent
2025-09-18 04:09:50,655 - DEBUG - Layer 2 path is seen for VM MAC 02:ea:cd:f4:82:1a in agent 10.0.0.27
2025-09-18 04:09:50,655 - DEBUG - Active layer 2 route in agent is present for VMI tapeacdf482-1a
2025-09-18 04:09:50,655 - DEBUG - L2 label(29) matches bw route table and interface table
2025-09-18 04:09:50,663 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is found in Agent of node 10.0.0.61
2025-09-18 04:09:50,794 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is consistent in agent 10.0.0.61
2025-09-18 04:09:50,795 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.61
2025-09-18 04:09:50,795 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 verification for VM ctest-vn1_vm2-74398981 in Agent 10.0.0.61 passed
2025-09-18 04:09:50,910 - DEBUG - Starting all layer 2 verification in agent 10.0.0.61
2025-09-18 04:09:50,924 - DEBUG - Route for VM MAC 02:ea:cd:f4:82:1a is consistent in agent 10.0.0.61
2025-09-18 04:09:50,933 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is found in Agent of node 10.0.0.27
2025-09-18 04:09:51,067 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 is consistent in agent 10.0.0.27
2025-09-18 04:09:51,067 - DEBUG - Route for VM IP ['192.168.1.4'] is consistent in agent 10.0.0.27
2025-09-18 04:09:51,067 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 verification for VM ctest-vn1_vm2-74398981 in Agent 10.0.0.27 passed
2025-09-18 04:09:51,184 - DEBUG - Starting all layer 2 verification in agent 10.0.0.27
2025-09-18 04:09:51,197 - DEBUG - Route for VM MAC 02:ea:cd:f4:82:1a is consistent in agent 10.0.0.27
2025-09-18 04:09:51,198 - INFO - VM ctest-vn1_vm2-74398981 verifications in Compute nodes passed
2025-09-18 04:09:51,447 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.61
2025-09-18 04:09:51,453 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: 10.20.0.192, 24; Got 10.20.0.192,24
2025-09-18 04:09:51,453 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.61
2025-09-18 04:09:51,569 - DEBUG - Validated VM route 192.168.1.4 in vrouter of 10.0.0.27
2025-09-18 04:09:51,575 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0
2025-09-18 04:09:51,575 - INFO - Validated routes of VM ctest-vn1_vm2-74398981 in all vrouters
2025-09-18 04:09:52,082 - DEBUG - Starting all layer2 verification in 10.0.0.22 Control Node
2025-09-18 04:09:52,221 - DEBUG - Layer2 route found for VM MAC 02:ea:cd:f4:82:1a in Control-node 10.0.0.22
2025-09-18 04:09:52,221 - DEBUG - L2 Label for VM ctest-vn1_vm2-74398981 same between Control-node 10.0.0.22 and Agent, Expected: 29, Seen: 29
2025-09-18 04:09:52,221 - DEBUG - Starting all layer2 verification in 10.0.0.41 Control Node
2025-09-18 04:09:52,353 - DEBUG - Layer2 route found for VM MAC 02:ea:cd:f4:82:1a in Control-node 10.0.0.41
2025-09-18 04:09:52,353 - DEBUG - L2 Label for VM ctest-vn1_vm2-74398981 same between Control-node 10.0.0.41 and Agent, Expected: 29, Seen: 29
2025-09-18 04:09:52,353 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-74398981 passed
2025-09-18 04:09:52,353 - DEBUG - Verifying the vm in opserver
2025-09-18 04:09:52,353 - DEBUG - Verifying in collector 10.0.0.41 ...
2025-09-18 04:09:52,353 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/7394bd4a-d6be-49f9-b845-c47c148de115?flat
2025-09-18 04:09:52,370 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-22347485:eacdf482-1a52-439d-b69d-7164b3ca6e53?flat
2025-09-18 04:09:52,396 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-22347485:eacdf482-1a52-439d-b69d-7164b3ca6e53?flat
2025-09-18 04:09:52,418 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/7394bd4a-d6be-49f9-b845-c47c148de115?flat
2025-09-18 04:09:52,438 - DEBUG - Verifying vm in vn uve
2025-09-18 04:09:52,439 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-22347485:eacdf482-1a52-439d-b69d-7164b3ca6e53?flat
2025-09-18 04:09:52,462 - DEBUG - VM uve shows interface as {'vm_name': 'ctest-vn1_vm2-74398981', 'is_health_check_active': True, 'ip6_address': '::', 'l2_active': True, 'rx_vlan': 65535, 'ip_address': '192.168.1.4', 'vm_uuid': '7394bd4a-d6be-49f9-b845-c47c148de115', 'active': True, 'fixed_ip4_list': ['192.168.1.4'], 'vhostuser_mode': 0, 'tx_vlan': 65535, 'admin_state': True, 'ip6_active': False, 'gateway': '192.168.1.1', 'uuid': 'eacdf482-1a52-439d-b69d-7164b3ca6e53', 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685', '__T': 1758168462484721, 'label': 24, 'ip4_active': True, 'mac_address': '02:ea:cd:f4:82:1a', 'vn_uuid': '3a2178fe-5c50-4c6e-8b9a-d91af4303325', 'port_mirror_enabled': False}
2025-09-18 04:09:52,462 - DEBUG - VM uve shows ip address as ['192.168.1.4', '::']
2025-09-18 04:09:52,462 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685
2025-09-18 04:09:52,462 - INFO - Verifying through opserver in 10.0.0.56
2025-09-18 04:09:52,462 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685?flat
2025-09-18 04:09:52,497 - DEBUG - expected vm list 7394bd4a-d6be-49f9-b845-c47c148de115
2025-09-18 04:09:52,497 - DEBUG - Extracted vm list ['7394bd4a-d6be-49f9-b845-c47c148de115']
2025-09-18 04:09:52,497 - INFO - VM 7394bd4a-d6be-49f9-b845-c47c148de115 is present in default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685
2025-09-18 04:09:52,497 - DEBUG - Verifying vm in vrouter uve
2025-09-18 04:09:52,498 - DEBUG - Getting info from collector 10.0.0.41..
2025-09-18 04:09:52,498 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/7394bd4a-d6be-49f9-b845-c47c148de115?flat
2025-09-18 04:09:52,514 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4130-2']
2025-09-18 04:09:52,514 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:09:52,568 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:09:52,605 - DEBUG - VM 7394bd4a-d6be-49f9-b845-c47c148de115 is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:09:52,605 - DEBUG - Validated that VM ctest-vn1_vm2-74398981 is in Vrouter an-jenkins-deploy-platform-ansible-os-4130-2 UVE
2025-09-18 04:09:52,605 - DEBUG - Verifying vm tap interface/vn in vrouter uve
2025-09-18 04:09:52,614 - DEBUG - Expected tap interface of VM uuid 7394bd4a-d6be-49f9-b845-c47c148de115 is default-domain:ctest-TestBasicVMVN0-22347485:eacdf482-1a52-439d-b69d-7164b3ca6e53
2025-09-18 04:09:52,614 - DEBUG - Expected VN of VM uuid 7394bd4a-d6be-49f9-b845-c47c148de115 is default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685
2025-09-18 04:09:52,614 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:09:52,680 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:09:52,725 - DEBUG - VM 7394bd4a-d6be-49f9-b845-c47c148de115 is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:09:52,725 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-22347485:eacdf482-1a52-439d-b69d-7164b3ca6e53 of vm 7394bd4a-d6be-49f9-b845-c47c148de115 is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:09:52,725 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 of vm 7394bd4a-d6be-49f9-b845-c47c148de115 is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:09:52,725 - DEBUG - Verifying the 7394bd4a-d6be-49f9-b845-c47c148de115 virtual network link through opserver 10.0.0.56
2025-09-18 04:09:52,725 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines
2025-09-18 04:09:52,739 - DEBUG - VM link and name as {'name': '7394bd4a-d6be-49f9-b845-c47c148de115', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-machine/7394bd4a-d6be-49f9-b845-c47c148de115?flat'}
2025-09-18 04:09:52,739 - DEBUG - Verifying the 7394bd4a-d6be-49f9-b845-c47c148de115 virtual network link through opserver 10.0.0.22
2025-09-18 04:09:52,739 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-09-18 04:09:52,804 - DEBUG - VM link and name as {'name': '7394bd4a-d6be-49f9-b845-c47c148de115', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-machine/7394bd4a-d6be-49f9-b845-c47c148de115?flat'}
2025-09-18 04:09:52,804 - DEBUG - Verifying the 7394bd4a-d6be-49f9-b845-c47c148de115 virtual network link through opserver 10.0.0.41
2025-09-18 04:09:52,804 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines
2025-09-18 04:09:52,817 - DEBUG - VM link and name as {'name': '7394bd4a-d6be-49f9-b845-c47c148de115', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-machine/7394bd4a-d6be-49f9-b845-c47c148de115?flat'}
2025-09-18 04:09:52,817 - INFO - VM ctest-vn1_vm2-74398981 validations in Opserver passed
2025-09-18 04:09:52,817 - INFO - Checking the ping between the VM again
2025-09-18 04:09:52,817 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:09:52,817 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.4, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.61, gateway password: c0ntrail123
2025-09-18 04:09:52,818 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.4
2025-09-18 04:09:55,401 - DEBUG - PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_req=1 ttl=64 time=6.43 ms
64 bytes from 192.168.1.4: icmp_req=2 ttl=64 time=1.28 ms
64 bytes from 192.168.1.4: icmp_req=3 ttl=64 time=1.29 ms
--- 192.168.1.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.287/3.006/6.436/2.425 ms
2025-09-18 04:09:55,401 - INFO - Ping to IP 192.168.1.4 from VM ctest-vn1_vm1-56080873 passed
2025-09-18 04:09:55,401 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:09:55,401 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.1.3, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.27, gateway password: c0ntrail123
2025-09-18 04:09:55,402 - DEBUG - ping -s 56 -c 3 -W 1 192.168.1.3
2025-09-18 04:09:57,856 - DEBUG - PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_req=1 ttl=64 time=4.15 ms
64 bytes from 192.168.1.3: icmp_req=2 ttl=64 time=1.16 ms
64 bytes from 192.168.1.3: icmp_req=3 ttl=64 time=1.71 ms
--- 192.168.1.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.163/2.345/4.159/1.302 ms
2025-09-18 04:09:57,857 - INFO - Ping to IP 192.168.1.3 from VM ctest-vn1_vm2-74398981 passed
2025-09-18 04:09:57,857 - INFO - Deleting VM ctest-vn1_vm2-74398981
2025-09-18 04:09:57,947 - DEBUG - Verifying in api server 10.0.0.41
2025-09-18 04:09:57,947 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/7394bd4a-d6be-49f9-b845-c47c148de115
2025-09-18 04:09:57,957 - DEBUG - VM ID 7394bd4a-d6be-49f9-b845-c47c148de115 of VM ctest-vn1_vm2-74398981 is still found in API Server
2025-09-18 04:09:59,957 - DEBUG - Verifying in api server 10.0.0.41
2025-09-18 04:09:59,958 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/7394bd4a-d6be-49f9-b845-c47c148de115
2025-09-18 04:09:59,965 - DEBUG - Response Code: 404
2025-09-18 04:09:59,965 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/7394bd4a-d6be-49f9-b845-c47c148de115
2025-09-18 04:09:59,971 - DEBUG - Response Code: 404
2025-09-18 04:09:59,971 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/7394bd4a-d6be-49f9-b845-c47c148de115
2025-09-18 04:09:59,977 - DEBUG - Response Code: 404
2025-09-18 04:09:59,978 - INFO - VM ctest-vn1_vm2-74398981 is fully removed in API-Server
2025-09-18 04:10:00,134 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found
2025-09-18 04:10:00,319 - DEBUG - VM route 192.168.1.4 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 of compute 10.0.0.61
2025-09-18 04:10:00,326 - DEBUG - Validated that vrouter 10.0.0.61 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685
2025-09-18 04:10:00,327 - DEBUG - Validated that vrouter 10.0.0.27 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685
2025-09-18 04:10:00,327 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685
2025-09-18 04:10:00,327 - INFO - VM ctest-vn1_vm2-74398981 is removed in Compute, and routes are removed in all compute nodes
2025-09-18 04:10:00,598 - INFO - Routes for VM ctest-vn1_vm2-74398981 is removed in all control-nodes
2025-09-18 04:10:00,598 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:10:00,665 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:10:00,722 - DEBUG - VM 7394bd4a-d6be-49f9-b845-c47c148de115 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:10:00,722 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:10:00,722 - DEBUG - interface for vm 7394bd4a-d6be-49f9-b845-c47c148de115 deleted from vrouter uve ...
2025-09-18 04:10:00,722 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:10:03,723 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:10:03,793 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:10:03,837 - DEBUG - VM 7394bd4a-d6be-49f9-b845-c47c148de115 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:10:03,837 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:10:03,837 - DEBUG - interface for vm 7394bd4a-d6be-49f9-b845-c47c148de115 deleted from vrouter uve ...
2025-09-18 04:10:03,837 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:10:06,838 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:10:06,906 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:10:06,946 - DEBUG - VM 7394bd4a-d6be-49f9-b845-c47c148de115 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:10:06,946 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:10:06,946 - DEBUG - interface for vm 7394bd4a-d6be-49f9-b845-c47c148de115 deleted from vrouter uve ...
2025-09-18 04:10:06,946 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:10:09,947 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:10:10,006 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:10:10,043 - DEBUG - VM 7394bd4a-d6be-49f9-b845-c47c148de115 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:10:10,043 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:10:10,043 - DEBUG - interface for vm 7394bd4a-d6be-49f9-b845-c47c148de115 deleted from vrouter uve ...
2025-09-18 04:10:10,043 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:10:13,044 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:10:13,100 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:10:13,137 - DEBUG - VM 7394bd4a-d6be-49f9-b845-c47c148de115 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:10:13,137 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:10:13,137 - DEBUG - interface for vm 7394bd4a-d6be-49f9-b845-c47c148de115 deleted from vrouter uve ...
2025-09-18 04:10:13,137 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:10:13,137 - DEBUG - Validated that VM 7394bd4a-d6be-49f9-b845-c47c148de115 is removed in Vrouter UVE
2025-09-18 04:10:13,137 - DEBUG - Verifying through opserver in 10.0.0.56
2025-09-18 04:10:13,137 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685?flat
2025-09-18 04:10:13,165 - DEBUG - 7394bd4a-d6be-49f9-b845-c47c148de115 BM not in default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 uve
2025-09-18 04:10:13,165 - DEBUG - Validated that VM 7394bd4a-d6be-49f9-b845-c47c148de115 is not present in the VN UVE
2025-09-18 04:10:13,165 - DEBUG - Verifying the 7394bd4a-d6be-49f9-b845-c47c148de115 virtual network link through opserver 10.0.0.56
2025-09-18 04:10:13,165 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines
2025-09-18 04:10:13,179 - INFO - 7394bd4a-d6be-49f9-b845-c47c148de115 vm uve deleted from opserver
2025-09-18 04:10:13,180 - DEBUG - Verifying the 7394bd4a-d6be-49f9-b845-c47c148de115 virtual network link through opserver 10.0.0.22
2025-09-18 04:10:13,180 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-09-18 04:10:13,253 - INFO - 7394bd4a-d6be-49f9-b845-c47c148de115 vm uve deleted from opserver
2025-09-18 04:10:13,253 - DEBUG - Verifying the 7394bd4a-d6be-49f9-b845-c47c148de115 virtual network link through opserver 10.0.0.41
2025-09-18 04:10:13,254 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines
2025-09-18 04:10:13,267 - INFO - 7394bd4a-d6be-49f9-b845-c47c148de115 vm uve deleted from opserver
2025-09-18 04:10:13,267 - INFO - Validated that VM 7394bd4a-d6be-49f9-b845-c47c148de115 is removed from Opserver
2025-09-18 04:10:13,267 - INFO - Deleting VM ctest-vn1_vm1-56080873
2025-09-18 04:10:13,362 - DEBUG - Verifying in api server 10.0.0.41
2025-09-18 04:10:13,363 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316
2025-09-18 04:10:13,373 - DEBUG - VM ID 4969978e-36f9-4abc-8e76-93e225439316 of VM ctest-vn1_vm1-56080873 is still found in API Server
2025-09-18 04:10:15,374 - DEBUG - Verifying in api server 10.0.0.41
2025-09-18 04:10:15,374 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316
2025-09-18 04:10:15,382 - DEBUG - Response Code: 404
2025-09-18 04:10:15,382 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316
2025-09-18 04:10:15,389 - DEBUG - Response Code: 404
2025-09-18 04:10:15,389 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/4969978e-36f9-4abc-8e76-93e225439316
2025-09-18 04:10:15,396 - DEBUG - Response Code: 404
2025-09-18 04:10:15,396 - INFO - VM ctest-vn1_vm1-56080873 is fully removed in API-Server
2025-09-18 04:10:15,560 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found
2025-09-18 04:10:15,612 - DEBUG - Validated that vrouter 10.0.0.61 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685
2025-09-18 04:10:15,625 - DEBUG - Validated that vrouter 10.0.0.27 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685
2025-09-18 04:10:15,625 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685
2025-09-18 04:10:15,626 - INFO - VM ctest-vn1_vm1-56080873 is removed in Compute, and routes are removed in all compute nodes
2025-09-18 04:10:15,874 - INFO - Routes for VM ctest-vn1_vm1-56080873 is removed in all control-nodes
2025-09-18 04:10:15,874 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:10:15,946 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:10:15,994 - DEBUG - VM 4969978e-36f9-4abc-8e76-93e225439316 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:10:15,994 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:10:15,994 - DEBUG - interface for vm 4969978e-36f9-4abc-8e76-93e225439316 deleted from vrouter uve ...
2025-09-18 04:10:15,994 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:10:18,995 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:10:19,062 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:10:19,122 - DEBUG - VM 4969978e-36f9-4abc-8e76-93e225439316 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:10:19,122 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:10:19,122 - DEBUG - interface for vm 4969978e-36f9-4abc-8e76-93e225439316 deleted from vrouter uve ...
2025-09-18 04:10:19,123 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:10:22,124 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:10:22,181 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:10:22,219 - DEBUG - VM 4969978e-36f9-4abc-8e76-93e225439316 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:10:22,219 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:10:22,219 - DEBUG - interface for vm 4969978e-36f9-4abc-8e76-93e225439316 deleted from vrouter uve ...
2025-09-18 04:10:22,219 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:10:25,220 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:10:25,283 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:10:25,320 - DEBUG - VM 4969978e-36f9-4abc-8e76-93e225439316 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:10:25,320 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:10:25,320 - DEBUG - interface for vm 4969978e-36f9-4abc-8e76-93e225439316 deleted from vrouter uve ...
2025-09-18 04:10:25,320 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:10:28,321 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:10:28,381 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:10:28,421 - DEBUG - VM 4969978e-36f9-4abc-8e76-93e225439316 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:10:28,421 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:10:28,421 - DEBUG - interface for vm 4969978e-36f9-4abc-8e76-93e225439316 deleted from vrouter uve ...
2025-09-18 04:10:28,421 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:10:31,422 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:10:31,476 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:10:31,513 - DEBUG - VM 4969978e-36f9-4abc-8e76-93e225439316 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:10:31,513 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:10:31,513 - DEBUG - interface for vm 4969978e-36f9-4abc-8e76-93e225439316 deleted from vrouter uve ...
2025-09-18 04:10:31,513 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:10:34,514 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:10:34,569 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:10:34,604 - DEBUG - VM 4969978e-36f9-4abc-8e76-93e225439316 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:10:34,604 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:10:34,604 - DEBUG - interface for vm 4969978e-36f9-4abc-8e76-93e225439316 deleted from vrouter uve ...
2025-09-18 04:10:34,604 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:10:34,604 - DEBUG - Validated that VM 4969978e-36f9-4abc-8e76-93e225439316 is removed in Vrouter UVE
2025-09-18 04:10:34,604 - DEBUG - Verifying through opserver in 10.0.0.56
2025-09-18 04:10:34,604 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685?flat
2025-09-18 04:10:34,623 - DEBUG - 4969978e-36f9-4abc-8e76-93e225439316 BM not in default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-28807685 uve
2025-09-18 04:10:34,623 - DEBUG - Validated that VM 4969978e-36f9-4abc-8e76-93e225439316 is not present in the VN UVE
2025-09-18 04:10:34,623 - DEBUG - Verifying the 4969978e-36f9-4abc-8e76-93e225439316 virtual network link through opserver 10.0.0.56
2025-09-18 04:10:34,623 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines
2025-09-18 04:10:34,637 - INFO - 4969978e-36f9-4abc-8e76-93e225439316 vm uve deleted from opserver
2025-09-18 04:10:34,637 - DEBUG - Verifying the 4969978e-36f9-4abc-8e76-93e225439316 virtual network link through opserver 10.0.0.22
2025-09-18 04:10:34,637 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-09-18 04:10:34,651 - INFO - 4969978e-36f9-4abc-8e76-93e225439316 vm uve deleted from opserver
2025-09-18 04:10:34,651 - DEBUG - Verifying the 4969978e-36f9-4abc-8e76-93e225439316 virtual network link through opserver 10.0.0.41
2025-09-18 04:10:34,651 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines
2025-09-18 04:10:34,664 - INFO - 4969978e-36f9-4abc-8e76-93e225439316 vm uve deleted from opserver
2025-09-18 04:10:34,664 - INFO - Validated that VM 4969978e-36f9-4abc-8e76-93e225439316 is removed from Opserver
2025-09-18 04:10:34,664 - INFO - Deleting VN ctest-vn1-28807685
2025-09-18 04:10:34,830 - DEBUG - Response for deleting network ()
2025-09-18 04:10:34,830 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/652b3292-3160-49c9-8985-481dd28e8beb
2025-09-18 04:10:34,837 - DEBUG - Response Code: 404
2025-09-18 04:10:34,837 - DEBUG - Requesting: http://10.0.0.56:8082/domains
2025-09-18 04:10:34,845 - DEBUG - Requesting: http://10.0.0.56:8082/domain/245899c5-19ba-4930-be6a-859f116860e1
2025-09-18 04:10:34,930 - DEBUG - Requesting: http://10.0.0.56:8082/project/0a1e7454-93bc-4746-b98c-6e3a6282a39d
2025-09-18 04:10:35,041 - INFO - Validated that VN ctest-vn1-28807685 is not found in API Server
2025-09-18 04:10:35,056 - DEBUG - VN ctest-vn1-28807685 is not present in Agent 10.0.0.61
2025-09-18 04:10:35,068 - DEBUG - VN ctest-vn1-28807685 is not present in Agent 10.0.0.27
2025-09-18 04:10:35,068 - INFO - Validated that VN ctest-vn1-28807685 is not in any agent
2025-09-18 04:10:35,108 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-28807685 info
2025-09-18 04:10:36,372 - DEBUG - Skipping xmpp flap check
2025-09-18 04:10:36,372 - INFO - END TEST : test_control_node_switchover : PASSED[0:03:23]
2025-09-18 04:10:36,372 - INFO - --------------------------------------------------------------------------------
2025-09-18 04:10:36,375 - INFO - ================================================================================
2025-09-18 04:10:36,375 - INFO - STARTING TEST : test_process_restart_in_policy_between_vns
2025-09-18 04:10:36,376 - INFO - TEST DESCRIPTION : Test to validate that with policy having rule to check icmp fwding between VMs on different VNs , ping between VMs should pass
with process restarts
1. Pick 2 VN's from resource pool which has one VM each
2. Create policy with icmp allow rule between those VN's and bind it networks
3. Ping from one VM to another VM
4. Restart process 'vrouter' and 'control' on setup
5. Ping again between VM's after process restart
Pass criteria: Step 2,3,4 and 5 should pass
2025-09-18 04:10:37,636 - DEBUG - Skipping xmpp flap check
2025-09-18 04:10:37,636 - INFO - Initial checks done. Running the testcase now
2025-09-18 04:10:37,636 - INFO -
2025-09-18 04:10:37,643 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None]
2025-09-18 04:10:37,958 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None]
2025-09-18 04:10:38,558 - INFO - Created VN ctest-vn1-44999393, UUID :6c7b9b58-48e3-40ad-806a-5420ca267106
2025-09-18 04:10:38,896 - DEBUG - Requesting: http://10.0.0.56:8082/domains
2025-09-18 04:10:38,904 - DEBUG - Requesting: http://10.0.0.56:8082/domain/245899c5-19ba-4930-be6a-859f116860e1
2025-09-18 04:10:38,918 - DEBUG - Requesting: http://10.0.0.56:8082/project/0a1e7454-93bc-4746-b98c-6e3a6282a39d
2025-09-18 04:10:39,050 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/6c7b9b58-48e3-40ad-806a-5420ca267106
2025-09-18 04:10:39,067 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/6c7b9b58-48e3-40ad-806a-5420ca267106
2025-09-18 04:10:39,080 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/47ac12e7-92e7-45d0-8989-7225debe3778
2025-09-18 04:10:39,090 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/47ac12e7-92e7-45d0-8989-7225debe3778
2025-09-18 04:10:39,100 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/d6c09707-87e8-4b59-8e2c-ceeb28f61504
2025-09-18 04:10:39,113 - DEBUG - Route Targets: ['target:64512:8000004']
2025-09-18 04:10:39,114 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/6c7b9b58-48e3-40ad-806a-5420ca267106
2025-09-18 04:10:39,128 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/47ac12e7-92e7-45d0-8989-7225debe3778
2025-09-18 04:10:39,154 - INFO - Verified VN network id 12 for VN 6c7b9b58-48e3-40ad-806a-5420ca267106
2025-09-18 04:10:39,154 - INFO - Verifications in API Server for VN ctest-vn1-44999393 passed
2025-09-18 04:10:39,154 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/6c7b9b58-48e3-40ad-806a-5420ca267106
2025-09-18 04:10:39,169 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/47ac12e7-92e7-45d0-8989-7225debe3778
2025-09-18 04:10:39,180 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/47ac12e7-92e7-45d0-8989-7225debe3778
2025-09-18 04:10:39,191 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/d6c09707-87e8-4b59-8e2c-ceeb28f61504
2025-09-18 04:10:39,212 - DEBUG - Control-node 10.0.0.56 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-22347485', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7817012381713907885', 'uuid-lslong': '9253300884193308934'}, 'enable': 'true', 'created': '2025-09-18T04:10:38', 'last-modified': '2025-09-18T04:10:38', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.509689'}
2025-09-18 04:10:39,220 - DEBUG - Route Targets: ['target:64512:8000004']
2025-09-18 04:10:39,226 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-22347485', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7817012381713907885', 'uuid-lslong': '9253300884193308934'}, 'enable': 'true', 'created': '2025-09-18T04:10:38', 'last-modified': '2025-09-18T04:10:38', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.524139'}
2025-09-18 04:10:39,232 - DEBUG - Route Targets: ['target:64512:8000004']
2025-09-18 04:10:39,239 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-22347485', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7817012381713907885', 'uuid-lslong': '9253300884193308934'}, 'enable': 'true', 'created': '2025-09-18T04:10:38', 'last-modified': '2025-09-18T04:10:38', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.538406'}
2025-09-18 04:10:39,247 - DEBUG - Route Targets: ['target:64512:8000004']
2025-09-18 04:10:39,247 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-44999393 passed
2025-09-18 04:10:39,247 - DEBUG - ====Verifying policy data for ctest-vn1-44999393 in API_Server ======
2025-09-18 04:10:39,247 - DEBUG - Requesting: http://10.0.0.56:8082/domains
2025-09-18 04:10:39,256 - DEBUG - Requesting: http://10.0.0.56:8082/domain/245899c5-19ba-4930-be6a-859f116860e1
2025-09-18 04:10:39,277 - DEBUG - Requesting: http://10.0.0.56:8082/project/0a1e7454-93bc-4746-b98c-6e3a6282a39d
2025-09-18 04:10:39,377 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/6c7b9b58-48e3-40ad-806a-5420ca267106
2025-09-18 04:10:39,397 - DEBUG - =>VN ctest-vn1-44999393 has no policy to be verified
2025-09-18 04:10:39,397 - DEBUG - Verifying the vn in opserver
2025-09-18 04:10:39,397 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 virtual network link through opserver 10.0.0.56
2025-09-18 04:10:39,397 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-networks
2025-09-18 04:10:39,412 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393?flat'}
2025-09-18 04:10:39,412 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 virtual network link through opserver 10.0.0.22
2025-09-18 04:10:39,412 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks
2025-09-18 04:10:39,434 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393?flat'}
2025-09-18 04:10:39,434 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 virtual network link through opserver 10.0.0.41
2025-09-18 04:10:39,434 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks
2025-09-18 04:10:39,448 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393?flat'}
2025-09-18 04:10:39,448 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is found in opserver
2025-09-18 04:10:39,478 - DEBUG - Do not have enough data to verify VN in agent
2025-09-18 04:10:39,491 - DEBUG - VRF ids for VN ctest-vn1-44999393: {}
2025-09-18 04:10:40,189 - INFO - Created VN ctest-vn2-33833937, UUID :81b38e96-1391-45f9-9c28-5b67dd101541
2025-09-18 04:10:40,522 - DEBUG - Requesting: http://10.0.0.56:8082/domains
2025-09-18 04:10:40,532 - DEBUG - Requesting: http://10.0.0.56:8082/domain/245899c5-19ba-4930-be6a-859f116860e1
2025-09-18 04:10:40,549 - DEBUG - Requesting: http://10.0.0.56:8082/project/0a1e7454-93bc-4746-b98c-6e3a6282a39d
2025-09-18 04:10:40,656 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/81b38e96-1391-45f9-9c28-5b67dd101541
2025-09-18 04:10:40,672 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/81b38e96-1391-45f9-9c28-5b67dd101541
2025-09-18 04:10:40,684 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/ebaa7738-37be-48f5-920d-e73acd625479
2025-09-18 04:10:40,698 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/ebaa7738-37be-48f5-920d-e73acd625479
2025-09-18 04:10:40,712 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/05c128c4-f787-4a1f-b81d-481c4b83cea1
2025-09-18 04:10:40,722 - DEBUG - Route Targets: ['target:64512:8000005']
2025-09-18 04:10:40,722 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/81b38e96-1391-45f9-9c28-5b67dd101541
2025-09-18 04:10:40,738 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/ebaa7738-37be-48f5-920d-e73acd625479
2025-09-18 04:10:40,759 - INFO - Verified VN network id 13 for VN 81b38e96-1391-45f9-9c28-5b67dd101541
2025-09-18 04:10:40,759 - INFO - Verifications in API Server for VN ctest-vn2-33833937 passed
2025-09-18 04:10:40,760 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/81b38e96-1391-45f9-9c28-5b67dd101541
2025-09-18 04:10:40,776 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/ebaa7738-37be-48f5-920d-e73acd625479
2025-09-18 04:10:40,788 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/ebaa7738-37be-48f5-920d-e73acd625479
2025-09-18 04:10:40,798 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/05c128c4-f787-4a1f-b81d-481c4b83cea1
2025-09-18 04:10:40,818 - DEBUG - Control-node 10.0.0.56 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-22347485', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9345970426948437497', 'uuid-lslong': '11252344170633762113'}, 'enable': 'true', 'created': '2025-09-18T04:10:40', 'last-modified': '2025-09-18T04:10:40', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.488437'}
2025-09-18 04:10:40,827 - DEBUG - Route Targets: ['target:64512:8000005']
2025-09-18 04:10:40,833 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-22347485', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9345970426948437497', 'uuid-lslong': '11252344170633762113'}, 'enable': 'true', 'created': '2025-09-18T04:10:40', 'last-modified': '2025-09-18T04:10:40', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.499435'}
2025-09-18 04:10:40,840 - DEBUG - Route Targets: ['target:64512:8000005']
2025-09-18 04:10:40,846 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-22347485', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9345970426948437497', 'uuid-lslong': '11252344170633762113'}, 'enable': 'true', 'created': '2025-09-18T04:10:40', 'last-modified': '2025-09-18T04:10:40', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.517820'}
2025-09-18 04:10:40,853 - DEBUG - Route Targets: ['target:64512:8000005']
2025-09-18 04:10:40,853 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-33833937 passed
2025-09-18 04:10:40,853 - DEBUG - ====Verifying policy data for ctest-vn2-33833937 in API_Server ======
2025-09-18 04:10:40,854 - DEBUG - Requesting: http://10.0.0.56:8082/domains
2025-09-18 04:10:40,861 - DEBUG - Requesting: http://10.0.0.56:8082/domain/245899c5-19ba-4930-be6a-859f116860e1
2025-09-18 04:10:40,874 - DEBUG - Requesting: http://10.0.0.56:8082/project/0a1e7454-93bc-4746-b98c-6e3a6282a39d
2025-09-18 04:10:40,981 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/81b38e96-1391-45f9-9c28-5b67dd101541
2025-09-18 04:10:40,993 - DEBUG - =>VN ctest-vn2-33833937 has no policy to be verified
2025-09-18 04:10:40,993 - DEBUG - Verifying the vn in opserver
2025-09-18 04:10:40,993 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 virtual network link through opserver 10.0.0.56
2025-09-18 04:10:40,993 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-networks
2025-09-18 04:10:41,011 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937?flat'}
2025-09-18 04:10:41,011 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 virtual network link through opserver 10.0.0.22
2025-09-18 04:10:41,011 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks
2025-09-18 04:10:41,024 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937?flat'}
2025-09-18 04:10:41,024 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 virtual network link through opserver 10.0.0.41
2025-09-18 04:10:41,024 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks
2025-09-18 04:10:41,037 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937?flat'}
2025-09-18 04:10:41,037 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 is found in opserver
2025-09-18 04:10:41,064 - DEBUG - Do not have enough data to verify VN in agent
2025-09-18 04:10:41,077 - DEBUG - VRF ids for VN ctest-vn2-33833937: {}
2025-09-18 04:10:42,828 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4130-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4130-1)
2025-09-18 04:10:44,418 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4130-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4130-2)
2025-09-18 04:10:44,418 - INFO - Waiting for VM ctest-vn1_vm1-99922935 to be up..
2025-09-18 04:10:44,531 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2025-09-18 04:10:49,618 - DEBUG - VM is in ACTIVE state now
2025-09-18 04:10:49,618 - INFO - VM name : ctest-vn1_vm1-99922935
2025-09-18 04:10:49,714 - DEBUG - VM ctest-vn1_vm1-99922935 ID is 0400f5cd-9875-4e49-9c86-315a521d84cd
2025-09-18 04:10:49,714 - DEBUG - VM ctest-vn1_vm1-99922935 launched on Node an-jenkins-deploy-platform-ansible-os-4130-1
2025-09-18 04:10:49,824 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/0400f5cd-9875-4e49-9c86-315a521d84cd
2025-09-18 04:10:49,837 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/59e92646-74f2-439f-9960-0e03b81bfbe6
2025-09-18 04:10:53,041 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms')
2025-09-18 04:10:53,041 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-99922935 failed!
2025-09-18 04:10:53,056 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:10:57,127 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms')
2025-09-18 04:10:57,127 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-99922935 failed!
2025-09-18 04:10:57,141 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:11:01,228 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms')
2025-09-18 04:11:01,228 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-99922935 failed!
2025-09-18 04:11:01,246 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:11:05,327 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms')
2025-09-18 04:11:05,328 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-99922935 failed!
2025-09-18 04:11:05,343 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:11:09,419 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms')
2025-09-18 04:11:09,419 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-99922935 failed!
2025-09-18 04:11:09,436 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:11:13,519 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms')
2025-09-18 04:11:13,519 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-99922935 failed!
2025-09-18 04:11:13,536 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:11:17,617 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms')
2025-09-18 04:11:17,617 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-99922935 failed!
2025-09-18 04:11:17,632 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:11:21,708 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms')
2025-09-18 04:11:21,708 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-99922935 failed!
2025-09-18 04:11:21,725 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:11:25,804 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms')
2025-09-18 04:11:25,805 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-99922935 failed!
2025-09-18 04:11:25,819 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:11:27,890 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=5.48 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=2.91 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 2.914/4.199/5.484/1.285 ms')
2025-09-18 04:11:27,890 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-99922935 passed
2025-09-18 04:11:27,962 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:11:27,962 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-99922935, IP 192.168.1.252, Port 22
2025-09-18 04:11:28,032 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-09-18 04:11:28,176 - DEBUG - VM ctest-vn1_vm1-99922935 is NOT ready for SSH connections, VM status: ACTIVE
2025-09-18 04:11:33,177 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:11:33,177 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-99922935, IP 192.168.1.252, Port 22
2025-09-18 04:11:33,243 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-09-18 04:11:33,336 - DEBUG - VM ctest-vn1_vm1-99922935 is NOT ready for SSH connections, VM status: ACTIVE
2025-09-18 04:11:38,337 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:11:38,337 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-99922935, IP 192.168.1.252, Port 22
2025-09-18 04:11:38,393 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-09-18 04:11:38,628 - DEBUG - VM ctest-vn1_vm1-99922935 is NOT ready for SSH connections, VM status: ACTIVE
2025-09-18 04:11:43,629 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:11:43,629 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-99922935, IP 192.168.1.252, Port 22
2025-09-18 04:11:43,686 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-09-18 04:11:43,782 - DEBUG - VM ctest-vn1_vm1-99922935 is NOT ready for SSH connections, VM status: ACTIVE
2025-09-18 04:11:48,782 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:11:48,783 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-99922935, IP 192.168.1.252, Port 22
2025-09-18 04:11:48,956 - DEBUG - VM ctest-vn1_vm1-99922935 is ready for SSH connections
2025-09-18 04:11:48,956 - INFO - Waiting for VM ctest-vn2_vm1-37053484 to be up..
2025-09-18 04:11:49,053 - DEBUG - VM is in ACTIVE state now
2025-09-18 04:11:49,053 - INFO - VM name : ctest-vn2_vm1-37053484
2025-09-18 04:11:49,147 - DEBUG - VM ctest-vn2_vm1-37053484 ID is a04c5bb1-ccf0-4f7a-affc-6d17f4a5102b
2025-09-18 04:11:49,147 - DEBUG - VM ctest-vn2_vm1-37053484 launched on Node an-jenkins-deploy-platform-ansible-os-4130-2
2025-09-18 04:11:49,235 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/a04c5bb1-ccf0-4f7a-affc-6d17f4a5102b
2025-09-18 04:11:49,247 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/b06272d7-0045-449a-b851-153226aebfc9
2025-09-18 04:11:50,417 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=13.2 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=2.36 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 2.356/7.790/13.225/5.434 ms')
2025-09-18 04:11:50,417 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm1-37053484 passed
2025-09-18 04:11:50,486 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:11:50,486 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-37053484, IP 192.168.2.252, Port 22
2025-09-18 04:11:50,544 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-09-18 04:11:50,645 - DEBUG - VM ctest-vn2_vm1-37053484 is NOT ready for SSH connections, VM status: ACTIVE
2025-09-18 04:11:55,646 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:11:55,646 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-37053484, IP 192.168.2.252, Port 22
2025-09-18 04:11:55,714 - DEBUG - Error on ssh to ubuntu@169.254.0.3:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-09-18 04:11:55,822 - DEBUG - VM ctest-vn2_vm1-37053484 is NOT ready for SSH connections, VM status: ACTIVE
2025-09-18 04:12:00,823 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:12:00,823 - DEBUG - Waiting to SSH to VM ctest-vn2_vm1-37053484, IP 192.168.2.252, Port 22
2025-09-18 04:12:00,991 - DEBUG - VM ctest-vn2_vm1-37053484 is ready for SSH connections
2025-09-18 04:12:00,991 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:12:00,991 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.252, host_string: ubuntu@169.254.0.3, password: ubuntugateway: ubuntu@10.0.0.61, gateway password: c0ntrail123
2025-09-18 04:12:00,991 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.252
2025-09-18 04:12:53,062 - DEBUG - PING 192.168.2.252 (192.168.2.252) 56(84) bytes of data.
64 bytes from 192.168.2.252: icmp_req=1 ttl=63 time=9.65 ms
64 bytes from 192.168.2.252: icmp_req=2 ttl=63 time=1.27 ms
64 bytes from 192.168.2.252: icmp_req=3 ttl=63 time=1.43 ms
--- 192.168.2.252 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.271/4.119/9.656/3.916 ms
2025-09-18 04:12:53,062 - INFO - Ping to IP 192.168.2.252 from VM ctest-vn1_vm1-99922935 passed
2025-09-18 04:12:56,485 - INFO - 10.0.0.61
2025-09-18 04:12:56,491 - INFO - agent:initializing
2025-09-18 04:12:56,492 - DEBUG - defaultdict(, {'10.0.0.61': {'agent': {'status': 'initializing', 'description': None}}})
2025-09-18 04:12:56,492 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-09-18 04:13:01,595 - INFO - 10.0.0.61
2025-09-18 04:13:01,603 - INFO - agent:active
2025-09-18 04:13:01,604 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.61
2025-09-18 04:13:04,974 - INFO - 10.0.0.27
2025-09-18 04:13:04,978 - INFO - agent:initializing
2025-09-18 04:13:04,978 - DEBUG - defaultdict(, {'10.0.0.27': {'agent': {'status': 'initializing', 'description': None}}})
2025-09-18 04:13:04,978 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-09-18 04:13:10,083 - INFO - 10.0.0.27
2025-09-18 04:13:10,091 - INFO - agent:active
2025-09-18 04:13:10,091 - INFO - Contrail services ['agent'] are up on nodes 10.0.0.27
2025-09-18 04:13:10,744 - INFO - 10.0.0.56
2025-09-18 04:13:10,747 - INFO - control:initializing
2025-09-18 04:13:10,747 - DEBUG - defaultdict(, {'10.0.0.56': {'control': {'status': 'initializing', 'description': None}}})
2025-09-18 04:13:10,747 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-09-18 04:13:15,860 - INFO - 10.0.0.56
2025-09-18 04:13:15,866 - INFO - control:active
2025-09-18 04:13:15,866 - INFO - Contrail services ['control'] are up on nodes 10.0.0.56
2025-09-18 04:13:16,269 - INFO - 10.0.0.22
2025-09-18 04:13:16,273 - INFO - control:initializing
2025-09-18 04:13:16,273 - DEBUG - defaultdict(, {'10.0.0.22': {'control': {'status': 'initializing', 'description': None}}})
2025-09-18 04:13:16,273 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-09-18 04:13:21,392 - INFO - 10.0.0.22
2025-09-18 04:13:21,399 - INFO - control:active
2025-09-18 04:13:21,399 - INFO - Contrail services ['control'] are up on nodes 10.0.0.22
2025-09-18 04:13:21,826 - INFO - 10.0.0.41
2025-09-18 04:13:21,833 - INFO - control:initializing
2025-09-18 04:13:21,834 - DEBUG - defaultdict(, {'10.0.0.41': {'control': {'status': 'initializing', 'description': None}}})
2025-09-18 04:13:21,835 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-09-18 04:13:26,934 - INFO - 10.0.0.41
2025-09-18 04:13:26,941 - INFO - control:active
2025-09-18 04:13:26,941 - INFO - Contrail services ['control'] are up on nodes 10.0.0.41
2025-09-18 04:13:27,846 - INFO - 10.0.0.56
2025-09-18 04:13:27,855 - INFO - api-server:initializing
2025-09-18 04:13:27,865 - DEBUG - defaultdict(, {'10.0.0.56': {'api-server': {'status': 'initializing', 'description': None}}})
2025-09-18 04:13:27,865 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-09-18 04:13:32,979 - INFO - 10.0.0.56
2025-09-18 04:13:32,985 - INFO - api-server:initializing
2025-09-18 04:13:32,985 - DEBUG - defaultdict(, {'10.0.0.56': {'api-server': {'status': 'initializing', 'description': None}}})
2025-09-18 04:13:32,985 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1
2025-09-18 04:13:38,107 - INFO - 10.0.0.56
2025-09-18 04:13:38,113 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down)
2025-09-18 04:13:38,114 - DEBUG - defaultdict(, {'10.0.0.56': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}})
2025-09-18 04:13:38,114 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2
2025-09-18 04:13:43,232 - INFO - 10.0.0.56
2025-09-18 04:13:43,237 - INFO - api-server:active
2025-09-18 04:13:43,237 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.56
2025-09-18 04:13:43,900 - INFO - 10.0.0.22
2025-09-18 04:13:43,905 - INFO - api-server:initializing
2025-09-18 04:13:43,905 - DEBUG - defaultdict(, {'10.0.0.22': {'api-server': {'status': 'initializing', 'description': None}}})
2025-09-18 04:13:43,905 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-09-18 04:13:49,004 - INFO - 10.0.0.22
2025-09-18 04:13:49,009 - INFO - api-server:initializing
2025-09-18 04:13:49,009 - DEBUG - defaultdict(, {'10.0.0.22': {'api-server': {'status': 'initializing', 'description': None}}})
2025-09-18 04:13:49,009 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1
2025-09-18 04:13:54,096 - INFO - 10.0.0.22
2025-09-18 04:13:54,102 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down)
2025-09-18 04:13:54,103 - DEBUG - defaultdict(, {'10.0.0.22': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}})
2025-09-18 04:13:54,103 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2
2025-09-18 04:13:59,193 - INFO - 10.0.0.22
2025-09-18 04:13:59,199 - INFO - api-server:active
2025-09-18 04:13:59,199 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.22
2025-09-18 04:13:59,958 - INFO - 10.0.0.41
2025-09-18 04:13:59,964 - INFO - api-server:initializing
2025-09-18 04:13:59,964 - DEBUG - defaultdict(, {'10.0.0.41': {'api-server': {'status': 'initializing', 'description': None}}})
2025-09-18 04:13:59,964 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 0
2025-09-18 04:14:05,063 - INFO - 10.0.0.41
2025-09-18 04:14:05,069 - INFO - api-server:initializing
2025-09-18 04:14:05,069 - DEBUG - defaultdict(, {'10.0.0.41': {'api-server': {'status': 'initializing', 'description': None}}})
2025-09-18 04:14:05,069 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 1
2025-09-18 04:14:10,167 - INFO - 10.0.0.41
2025-09-18 04:14:10,172 - INFO - api-server:initializing (Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down)
2025-09-18 04:14:10,173 - DEBUG - defaultdict(, {'10.0.0.41': {'api-server': {'status': 'initializing', 'description': 'Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down'}}})
2025-09-18 04:14:10,173 - DEBUG - Not all services up. Sleeping for 5 seconds. iteration: 2
2025-09-18 04:14:15,261 - INFO - 10.0.0.41
2025-09-18 04:14:15,266 - INFO - api-server:active
2025-09-18 04:14:15,266 - INFO - Contrail services ['api-server'] are up on nodes 10.0.0.41
2025-09-18 04:14:15,375 - INFO - 10.0.0.56
2025-09-18 04:14:15,375 - INFO - config
2025-09-18 04:14:15,381 - INFO - config-nodemgr:active
2025-09-18 04:14:15,387 - INFO - api-server:active
2025-09-18 04:14:15,391 - INFO - schema:backup
2025-09-18 04:14:15,394 - INFO - svc-monitor:backup
2025-09-18 04:14:15,398 - INFO - device-manager:backup
2025-09-18 04:14:15,398 - INFO - config-database
2025-09-18 04:14:15,398 - INFO - config-cassandra:active
2025-09-18 04:14:15,398 - INFO - config-zookeeper:active
2025-09-18 04:14:15,398 - INFO - config-rabbitmq:active
2025-09-18 04:14:15,398 - INFO - control
2025-09-18 04:14:15,406 - INFO - control-nodemgr:active
2025-09-18 04:14:15,412 - INFO - control:active
2025-09-18 04:14:15,413 - INFO - named:active
2025-09-18 04:14:15,418 - INFO - dns:active
2025-09-18 04:14:15,418 - INFO - analytics
2025-09-18 04:14:15,423 - INFO - analytics-nodemgr:active
2025-09-18 04:14:15,429 - INFO - analytics-api:active
2025-09-18 04:14:15,436 - INFO - collector:active
2025-09-18 04:14:15,436 - INFO - analytics-database
2025-09-18 04:14:15,436 - INFO - analytics-cassandra:active
2025-09-18 04:14:15,443 - INFO - analyticsdb-nodemgr:active
2025-09-18 04:14:15,449 - INFO - query-engine:active
2025-09-18 04:14:15,449 - INFO - webui
2025-09-18 04:14:15,450 - INFO - webui:active
2025-09-18 04:14:15,450 - INFO - webui-middleware:active
2025-09-18 04:14:15,450 - INFO - redis:active
2025-09-18 04:14:15,450 - INFO - analytics_snmp
2025-09-18 04:14:15,455 - INFO - snmp-collector:active
2025-09-18 04:14:15,455 - INFO - snmp-topology:active
2025-09-18 04:14:15,455 - INFO - analytics_alarm
2025-09-18 04:14:15,455 - INFO - alarmgen:active
2025-09-18 04:14:15,555 - INFO - 10.0.0.22
2025-09-18 04:14:15,555 - INFO - config
2025-09-18 04:14:15,563 - INFO - config-nodemgr:active
2025-09-18 04:14:15,568 - INFO - api-server:active
2025-09-18 04:14:15,572 - INFO - schema:backup
2025-09-18 04:14:15,576 - INFO - svc-monitor:active
2025-09-18 04:14:15,580 - INFO - device-manager:backup
2025-09-18 04:14:15,580 - INFO - config-database
2025-09-18 04:14:15,580 - INFO - config-cassandra:active
2025-09-18 04:14:15,580 - INFO - config-zookeeper:active
2025-09-18 04:14:15,580 - INFO - config-rabbitmq:active
2025-09-18 04:14:15,580 - INFO - control
2025-09-18 04:14:15,586 - INFO - control-nodemgr:active
2025-09-18 04:14:15,593 - INFO - control:active
2025-09-18 04:14:15,593 - INFO - named:active
2025-09-18 04:14:15,598 - INFO - dns:active
2025-09-18 04:14:15,598 - INFO - analytics
2025-09-18 04:14:15,604 - INFO - analytics-nodemgr:active
2025-09-18 04:14:15,609 - INFO - analytics-api:active
2025-09-18 04:14:15,614 - INFO - collector:active
2025-09-18 04:14:15,614 - INFO - analytics-database
2025-09-18 04:14:15,614 - INFO - analytics-cassandra:active
2025-09-18 04:14:15,620 - INFO - analyticsdb-nodemgr:active
2025-09-18 04:14:15,624 - INFO - query-engine:active
2025-09-18 04:14:15,625 - INFO - webui
2025-09-18 04:14:15,625 - INFO - webui:active
2025-09-18 04:14:15,625 - INFO - webui-middleware:active
2025-09-18 04:14:15,625 - INFO - redis:active
2025-09-18 04:14:15,625 - INFO - analytics_snmp
2025-09-18 04:14:15,629 - INFO - snmp-collector:active
2025-09-18 04:14:15,629 - INFO - snmp-topology:active
2025-09-18 04:14:15,629 - INFO - analytics_alarm
2025-09-18 04:14:15,629 - INFO - alarmgen:active
2025-09-18 04:14:15,729 - INFO - 10.0.0.41
2025-09-18 04:14:15,729 - INFO - config
2025-09-18 04:14:15,735 - INFO - config-nodemgr:active
2025-09-18 04:14:15,740 - INFO - api-server:active
2025-09-18 04:14:15,745 - INFO - schema:active
2025-09-18 04:14:15,749 - INFO - svc-monitor:backup
2025-09-18 04:14:15,753 - INFO - device-manager:active
2025-09-18 04:14:15,754 - INFO - config-database
2025-09-18 04:14:15,754 - INFO - config-cassandra:active
2025-09-18 04:14:15,754 - INFO - config-zookeeper:active
2025-09-18 04:14:15,754 - INFO - config-rabbitmq:active
2025-09-18 04:14:15,754 - INFO - control
2025-09-18 04:14:15,760 - INFO - control-nodemgr:active
2025-09-18 04:14:15,766 - INFO - control:active
2025-09-18 04:14:15,766 - INFO - named:active
2025-09-18 04:14:15,773 - INFO - dns:active
2025-09-18 04:14:15,773 - INFO - analytics
2025-09-18 04:14:15,780 - INFO - analytics-nodemgr:active
2025-09-18 04:14:15,784 - INFO - analytics-api:active
2025-09-18 04:14:15,791 - INFO - collector:active
2025-09-18 04:14:15,791 - INFO - analytics-database
2025-09-18 04:14:15,791 - INFO - analytics-cassandra:active
2025-09-18 04:14:15,796 - INFO - analyticsdb-nodemgr:active
2025-09-18 04:14:15,802 - INFO - query-engine:active
2025-09-18 04:14:15,802 - INFO - webui
2025-09-18 04:14:15,802 - INFO - webui:active
2025-09-18 04:14:15,802 - INFO - webui-middleware:active
2025-09-18 04:14:15,802 - INFO - redis:active
2025-09-18 04:14:15,802 - INFO - analytics_snmp
2025-09-18 04:14:15,808 - INFO - snmp-collector:active
2025-09-18 04:14:15,808 - INFO - snmp-topology:active
2025-09-18 04:14:15,808 - INFO - analytics_alarm
2025-09-18 04:14:15,808 - INFO - alarmgen:active
2025-09-18 04:14:15,908 - INFO - 10.0.0.61
2025-09-18 04:14:15,908 - INFO - vrouter
2025-09-18 04:14:15,914 - INFO - vrouter-nodemgr:active
2025-09-18 04:14:15,921 - INFO - agent:active
2025-09-18 04:14:16,021 - INFO - 10.0.0.27
2025-09-18 04:14:16,021 - INFO - vrouter
2025-09-18 04:14:16,028 - INFO - vrouter-nodemgr:active
2025-09-18 04:14:16,033 - INFO - agent:active
2025-09-18 04:14:16,033 - INFO - All the contrail services are up on all nodes
2025-09-18 04:14:16,033 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:14:16,125 - WARNING - an-jenkins-deploy-platform-ansible-os-4130-1:contrail-vrouter-agent module connection to ['10.20.0.25:5269'] servers NOT UP
2025-09-18 04:14:16,126 - INFO - an-jenkins-deploy-platform-ansible-os-4130-1:contrail-vrouter-agent module connection to ['10.20.0.17:5269'] servers UP
2025-09-18 04:14:16,126 - INFO - an-jenkins-deploy-platform-ansible-os-4130-1 connected to xmpp 10.20.0.17
2025-09-18 04:14:16,126 - INFO - an-jenkins-deploy-platform-ansible-os-4130-1:contrail-vrouter-agent module connection to ['10.20.0.14:5269'] servers UP
2025-09-18 04:14:16,126 - INFO - an-jenkins-deploy-platform-ansible-os-4130-1 connected to xmpp 10.20.0.14
2025-09-18 04:14:16,126 - INFO - an-jenkins-deploy-platform-ansible-os-4130-1:contrail-vrouter-agent module connection to ['10.20.0.25:53'] servers UP
2025-09-18 04:14:16,126 - INFO - an-jenkins-deploy-platform-ansible-os-4130-1 connected to dns 10.20.0.25
2025-09-18 04:14:16,126 - INFO - an-jenkins-deploy-platform-ansible-os-4130-1:contrail-vrouter-agent module connection to ['10.20.0.17:53'] servers UP
2025-09-18 04:14:16,126 - INFO - an-jenkins-deploy-platform-ansible-os-4130-1 connected to dns 10.20.0.17
2025-09-18 04:14:16,126 - WARNING - an-jenkins-deploy-platform-ansible-os-4130-1:contrail-vrouter-agent module connection to ['10.20.0.14:53'] servers NOT UP
2025-09-18 04:14:16,126 - WARNING - an-jenkins-deploy-platform-ansible-os-4130-1:contrail-vrouter-agent module connection to ['10.0.0.56:8086'] servers NOT UP
2025-09-18 04:14:16,126 - INFO - an-jenkins-deploy-platform-ansible-os-4130-1:contrail-vrouter-agent module connection to ['10.0.0.22:8086'] servers UP
2025-09-18 04:14:16,126 - INFO - an-jenkins-deploy-platform-ansible-os-4130-1 connected to collector 10.0.0.22
2025-09-18 04:14:16,126 - WARNING - an-jenkins-deploy-platform-ansible-os-4130-1:contrail-vrouter-agent module connection to ['10.0.0.41:8086'] servers NOT UP
2025-09-18 04:14:16,127 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:14:16,573 - INFO - cn-jenkins-deploy-platform-ansible-os-4130-1:contrail-control module connection to 10.0.0.56:9041 servers UP
2025-09-18 04:14:16,574 - INFO - cn-jenkins-deploy-platform-ansible-os-4130-1:contrail-control module connection to 10.0.0.56:5673 servers UP
2025-09-18 04:14:16,574 - INFO - cn-jenkins-deploy-platform-ansible-os-4130-1:contrail-control module connection to ['10.0.0.56:8086'] servers UP
2025-09-18 04:14:16,574 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:14:17,004 - INFO - cn-jenkins-deploy-platform-ansible-os-4130-2:contrail-control module connection to 10.0.0.56:9041 servers UP
2025-09-18 04:14:17,004 - INFO - cn-jenkins-deploy-platform-ansible-os-4130-2:contrail-control module connection to 10.0.0.56:5673 servers UP
2025-09-18 04:14:17,004 - INFO - cn-jenkins-deploy-platform-ansible-os-4130-2:contrail-control module connection to ['10.0.0.56:8086'] servers UP
2025-09-18 04:14:17,004 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-4130-3?flat
2025-09-18 04:14:17,027 - INFO - cn-jenkins-deploy-platform-ansible-os-4130-3:contrail-control module connection to 10.0.0.56:9041 servers UP
2025-09-18 04:14:17,027 - INFO - cn-jenkins-deploy-platform-ansible-os-4130-3:contrail-control module connection to 10.0.0.56:5673 servers UP
2025-09-18 04:14:17,027 - INFO - cn-jenkins-deploy-platform-ansible-os-4130-3:contrail-control module connection to ['10.0.0.56:8086'] servers UP
2025-09-18 04:14:17,027 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4130-1.?flat
2025-09-18 04:14:17,054 - DEBUG - contrail-api is running
2025-09-18 04:14:17,055 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4130-2.?flat
2025-09-18 04:14:17,080 - DEBUG - contrail-api is running
2025-09-18 04:14:17,080 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-4130-3.?flat
2025-09-18 04:14:17,109 - DEBUG - contrail-api is running
2025-09-18 04:14:17,421 - INFO - Created VN ctest-vn3-65676010, UUID :e446a1e5-b124-4d01-bd2d-e18fbf6f570c
2025-09-18 04:14:17,930 - DEBUG - Requesting: http://10.0.0.56:8082/domains
2025-09-18 04:14:17,985 - DEBUG - Requesting: http://10.0.0.56:8082/domain/245899c5-19ba-4930-be6a-859f116860e1
2025-09-18 04:14:18,002 - DEBUG - Requesting: http://10.0.0.56:8082/project/0a1e7454-93bc-4746-b98c-6e3a6282a39d
2025-09-18 04:14:18,126 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/6c7b9b58-48e3-40ad-806a-5420ca267106
2025-09-18 04:14:18,170 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/6c7b9b58-48e3-40ad-806a-5420ca267106
2025-09-18 04:14:18,182 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/47ac12e7-92e7-45d0-8989-7225debe3778
2025-09-18 04:14:18,192 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/47ac12e7-92e7-45d0-8989-7225debe3778
2025-09-18 04:14:18,201 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/d6c09707-87e8-4b59-8e2c-ceeb28f61504
2025-09-18 04:14:18,211 - DEBUG - Route Targets: ['target:64512:8000004']
2025-09-18 04:14:18,211 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/6c7b9b58-48e3-40ad-806a-5420ca267106
2025-09-18 04:14:18,224 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/47ac12e7-92e7-45d0-8989-7225debe3778
2025-09-18 04:14:18,247 - INFO - Verified VN network id 12 for VN 6c7b9b58-48e3-40ad-806a-5420ca267106
2025-09-18 04:14:18,247 - INFO - Verifications in API Server for VN ctest-vn1-44999393 passed
2025-09-18 04:14:18,247 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/6c7b9b58-48e3-40ad-806a-5420ca267106
2025-09-18 04:14:18,265 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/47ac12e7-92e7-45d0-8989-7225debe3778
2025-09-18 04:14:18,275 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/47ac12e7-92e7-45d0-8989-7225debe3778
2025-09-18 04:14:18,286 - DEBUG - Requesting: http://10.0.0.56:8082/route-target/d6c09707-87e8-4b59-8e2c-ceeb28f61504
2025-09-18 04:14:18,302 - DEBUG - Control-node 10.0.0.56 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-22347485', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7817012381713907885', 'uuid-lslong': '9253300884193308934'}, 'enable': 'true', 'created': '2025-09-18T04:10:38', 'last-modified': '2025-09-18T04:10:39', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:06.970953'}
2025-09-18 04:14:18,309 - DEBUG - Route Targets: ['target:64512:8000004']
2025-09-18 04:14:18,315 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-22347485', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7817012381713907885', 'uuid-lslong': '9253300884193308934'}, 'enable': 'true', 'created': '2025-09-18T04:10:38', 'last-modified': '2025-09-18T04:10:39', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:01.707196'}
2025-09-18 04:14:18,322 - DEBUG - Route Targets: ['target:64512:8000004']
2025-09-18 04:14:18,329 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestBasicVMVN0-22347485', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7817012381713907885', 'uuid-lslong': '9253300884193308934'}, 'enable': 'true', 'created': '2025-09-18T04:10:38', 'last-modified': '2025-09-18T04:10:39', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:56.065403'}
2025-09-18 04:14:18,336 - DEBUG - Route Targets: ['target:64512:8000004']
2025-09-18 04:14:18,336 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-44999393 passed
2025-09-18 04:14:18,336 - DEBUG - ====Verifying policy data for ctest-vn1-44999393 in API_Server ======
2025-09-18 04:14:18,337 - DEBUG - Requesting: http://10.0.0.56:8082/domains
2025-09-18 04:14:18,344 - DEBUG - Requesting: http://10.0.0.56:8082/domain/245899c5-19ba-4930-be6a-859f116860e1
2025-09-18 04:14:18,357 - DEBUG - Requesting: http://10.0.0.56:8082/project/0a1e7454-93bc-4746-b98c-6e3a6282a39d
2025-09-18 04:14:18,449 - DEBUG - Requesting: http://10.0.0.56:8082/virtual-network/6c7b9b58-48e3-40ad-806a-5420ca267106
2025-09-18 04:14:18,478 - DEBUG - ==>Verifying data for policy with id: f320a57c-7186-4479-a6d1-ce9597c4cf0f, fqn: ['default-domain', 'ctest-TestBasicVMVN0-22347485', 'policy1']
2025-09-18 04:14:18,478 - INFO - VN ctest-vn1-44999393 Policy verification: verify_vn_policy_in_api_server, status: True
2025-09-18 04:14:18,479 - DEBUG - Verifying the vn in opserver
2025-09-18 04:14:18,479 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 virtual network link through opserver 10.0.0.56
2025-09-18 04:14:18,479 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-networks
2025-09-18 04:14:18,499 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393?flat'}
2025-09-18 04:14:18,499 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 virtual network link through opserver 10.0.0.22
2025-09-18 04:14:18,499 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks
2025-09-18 04:14:18,617 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393?flat'}
2025-09-18 04:14:18,617 - DEBUG - Verifying the default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 virtual network link through opserver 10.0.0.41
2025-09-18 04:14:18,617 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks
2025-09-18 04:14:18,690 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393?flat'}
2025-09-18 04:14:18,690 - INFO - Validated that VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is found in opserver
2025-09-18 04:14:18,706 - DEBUG - VRF ids for VN ctest-vn1-44999393: {'10.0.0.61': '2'}
2025-09-18 04:14:20,156 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4130-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4130-1)
2025-09-18 04:14:20,215 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2025-09-18 04:14:25,308 - DEBUG - VM is in ACTIVE state now
2025-09-18 04:14:25,309 - INFO - VM name : ctest-vn1_vm2-10235123
2025-09-18 04:14:25,404 - DEBUG - VM ctest-vn1_vm2-10235123 ID is efa300aa-9dfd-4fef-8ce9-996faabf9f65
2025-09-18 04:14:25,506 - DEBUG - VM is in ACTIVE state now
2025-09-18 04:14:25,506 - DEBUG - VM ctest-vn1_vm2-10235123 launched on Node an-jenkins-deploy-platform-ansible-os-4130-1
2025-09-18 04:14:25,608 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65
2025-09-18 04:14:25,645 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65
2025-09-18 04:14:25,653 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/20405166-5e1b-4ebf-8642-8166a1e97b4b
2025-09-18 04:14:25,663 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65
2025-09-18 04:14:25,672 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/20405166-5e1b-4ebf-8642-8166a1e97b4b
2025-09-18 04:14:25,683 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/7f09830a-1128-41e3-9924-0d078ebfebdb
2025-09-18 04:14:25,695 - DEBUG - Verifying in api server 10.0.0.41
2025-09-18 04:14:25,695 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65
2025-09-18 04:14:25,710 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/20405166-5e1b-4ebf-8642-8166a1e97b4b
2025-09-18 04:14:25,721 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/7f09830a-1128-41e3-9924-0d078ebfebdb
2025-09-18 04:14:25,731 - INFO - VM ctest-vn1_vm2-10235123 verfication in all API Servers passed
2025-09-18 04:14:25,761 - DEBUG - VM ctest-vn1_vm2-10235123 Tap interface: {'index': '4', 'name': 'tap20405166-5e', 'uuid': '20405166-5e1b-4ebf-8642-8166a1e97b4b', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393:ctest-vn1-44999393', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'vm_uuid': 'efa300aa-9dfd-4fef-8ce9-996faabf9f65', 'vm_name': 'ctest-vn1_vm2-10235123', 'ip_addr': '192.168.1.251', 'mac_addr': '02:20:40:51:66:5e', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '167', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-22347485:20405166-5e1b-4ebf-8642-8166a1e97b4b', 'sg_uuid_list': ['f51220a7-8281-4ea3-8440-e34dbd8dc48d'], 'static_route_list': None, 'vm_project_uuid': '0a1e7454-93bc-4746-b98c-6e3a6282a39d', 'admin_state': 'Enabled', 'flow_key_idx': '28', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-09-18 04:14:25,761 - DEBUG - Agent 10.0.0.61 vrf name: default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393:ctest-vn1-44999393
2025-09-18 04:14:25,769 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393:ctest-vn1-44999393', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'table_label': '-1', 'vxlan_id': '12', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.129:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]}
2025-09-18 04:14:25,858 - DEBUG - VM ctest-vn1_vm2-10235123 : Tap interface tap20405166-5e is set to Active
2025-09-18 04:14:25,858 - DEBUG - Tap interface tap20405166-5e detail : {'index': '4', 'name': 'tap20405166-5e', 'uuid': '20405166-5e1b-4ebf-8642-8166a1e97b4b', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393:ctest-vn1-44999393', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '12', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'vm_uuid': 'efa300aa-9dfd-4fef-8ce9-996faabf9f65', 'vm_name': 'ctest-vn1_vm2-10235123', 'ip_addr': '192.168.1.251', 'mac_addr': '02:20:40:51:66:5e', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '167', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-22347485:20405166-5e1b-4ebf-8642-8166a1e97b4b', 'sg_uuid_list': ['f51220a7-8281-4ea3-8440-e34dbd8dc48d'], 'static_route_list': None, 'vm_project_uuid': '0a1e7454-93bc-4746-b98c-6e3a6282a39d', 'admin_state': 'Enabled', 'flow_key_idx': '28', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-09-18 04:14:25,971 - DEBUG - Starting Layer 2 verification in Agent
2025-09-18 04:14:25,986 - DEBUG - Layer 2 path is seen for VM MAC 02:20:40:51:66:5e in agent 10.0.0.61
2025-09-18 04:14:25,986 - DEBUG - Active layer 2 route in agent is present for VMI tap20405166-5e
2025-09-18 04:14:25,986 - DEBUG - L2 label(34) matches bw route table and interface table
2025-09-18 04:14:25,996 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is found in Agent of node 10.0.0.61
2025-09-18 04:14:26,139 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is consistent in agent 10.0.0.61
2025-09-18 04:14:26,140 - DEBUG - Route for VM IP ['192.168.1.251'] is consistent in agent 10.0.0.61
2025-09-18 04:14:26,140 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 verification for VM ctest-vn1_vm2-10235123 in Agent 10.0.0.61 passed
2025-09-18 04:14:26,269 - DEBUG - Starting all layer 2 verification in agent 10.0.0.61
2025-09-18 04:14:26,286 - DEBUG - Route for VM MAC 02:20:40:51:66:5e is consistent in agent 10.0.0.61
2025-09-18 04:14:29,489 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms')
2025-09-18 04:14:29,489 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-10235123 failed!
2025-09-18 04:14:29,505 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:14:33,582 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1009ms')
2025-09-18 04:14:33,582 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-10235123 failed!
2025-09-18 04:14:33,600 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:14:37,678 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms')
2025-09-18 04:14:37,679 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-10235123 failed!
2025-09-18 04:14:37,693 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:14:41,773 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms')
2025-09-18 04:14:41,773 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-10235123 failed!
2025-09-18 04:14:41,787 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:14:45,863 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms')
2025-09-18 04:14:45,863 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-10235123 failed!
2025-09-18 04:14:45,878 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:14:49,960 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms')
2025-09-18 04:14:49,960 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-10235123 failed!
2025-09-18 04:14:49,974 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:14:54,064 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1023ms')
2025-09-18 04:14:54,064 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-10235123 failed!
2025-09-18 04:14:54,079 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:14:58,160 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms')
2025-09-18 04:14:58,160 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-10235123 failed!
2025-09-18 04:14:58,175 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:15:02,248 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms')
2025-09-18 04:15:02,248 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-10235123 failed!
2025-09-18 04:15:02,265 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 is 192.168.1.254 and allocation pool is NOT set
2025-09-18 04:15:04,331 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=17.0 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=4.64 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 4.644/10.837/17.030/6.193 ms')
2025-09-18 04:15:04,332 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-10235123 passed
2025-09-18 04:15:04,332 - INFO - VM ctest-vn1_vm2-10235123 verifications in Compute nodes passed
2025-09-18 04:15:04,578 - DEBUG - Validated VM route 192.168.1.251 in vrouter of 10.0.0.61
2025-09-18 04:15:04,585 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0
2025-09-18 04:15:04,585 - INFO - Validated routes of VM ctest-vn1_vm2-10235123 in all vrouters
2025-09-18 04:15:05,028 - DEBUG - Starting all layer2 verification in 10.0.0.22 Control Node
2025-09-18 04:15:05,156 - DEBUG - Layer2 route found for VM MAC 02:20:40:51:66:5e in Control-node 10.0.0.22
2025-09-18 04:15:05,157 - DEBUG - L2 Label for VM ctest-vn1_vm2-10235123 same between Control-node 10.0.0.22 and Agent, Expected: 34, Seen: 34
2025-09-18 04:15:05,157 - DEBUG - Starting all layer2 verification in 10.0.0.41 Control Node
2025-09-18 04:15:05,285 - DEBUG - Layer2 route found for VM MAC 02:20:40:51:66:5e in Control-node 10.0.0.41
2025-09-18 04:15:05,286 - DEBUG - L2 Label for VM ctest-vn1_vm2-10235123 same between Control-node 10.0.0.41 and Agent, Expected: 34, Seen: 34
2025-09-18 04:15:05,286 - DEBUG - Starting all layer2 verification in 10.0.0.56 Control Node
2025-09-18 04:15:05,418 - DEBUG - Layer2 route found for VM MAC 02:20:40:51:66:5e in Control-node 10.0.0.56
2025-09-18 04:15:05,419 - DEBUG - L2 Label for VM ctest-vn1_vm2-10235123 same between Control-node 10.0.0.56 and Agent, Expected: 34, Seen: 34
2025-09-18 04:15:05,419 - INFO - Verification in Control-nodes for VM ctest-vn1_vm2-10235123 passed
2025-09-18 04:15:05,419 - DEBUG - Verifying the vm in opserver
2025-09-18 04:15:05,419 - DEBUG - Verifying in collector 10.0.0.41 ...
2025-09-18 04:15:05,419 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65?flat
2025-09-18 04:15:05,491 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-22347485:20405166-5e1b-4ebf-8642-8166a1e97b4b?flat
2025-09-18 04:15:05,515 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-22347485:20405166-5e1b-4ebf-8642-8166a1e97b4b?flat
2025-09-18 04:15:05,537 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65?flat
2025-09-18 04:15:05,614 - DEBUG - Verifying vm in vn uve
2025-09-18 04:15:05,614 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-22347485:20405166-5e1b-4ebf-8642-8166a1e97b4b?flat
2025-09-18 04:15:05,636 - DEBUG - VM uve shows interface as {'admin_state': True, 'is_health_check_active': True, 'fixed_ip4_list': ['192.168.1.251'], 'ip6_address': '::', 'tx_vlan': 65535, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393', 'ip6_active': False, 'port_mirror_enabled': False, 'uuid': '20405166-5e1b-4ebf-8642-8166a1e97b4b', 'ip4_active': True, 'vm_name': 'ctest-vn1_vm2-10235123', 'vm_uuid': 'efa300aa-9dfd-4fef-8ce9-996faabf9f65', 'l2_active': True, 'mac_address': '02:20:40:51:66:5e', 'ip_address': '192.168.1.251', 'rx_vlan': 65535, 'vn_uuid': '6c7b9b58-48e3-40ad-806a-5420ca267106', 'active': True, '__T': 1758168867178041, 'vhostuser_mode': 0, 'gateway': '192.168.1.254', 'label': 30}
2025-09-18 04:15:05,636 - DEBUG - VM uve shows ip address as ['192.168.1.251', '::']
2025-09-18 04:15:05,636 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393
2025-09-18 04:15:05,636 - INFO - Verifying through opserver in 10.0.0.56
2025-09-18 04:15:05,636 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393?flat
2025-09-18 04:15:05,664 - DEBUG - expected vm list efa300aa-9dfd-4fef-8ce9-996faabf9f65
2025-09-18 04:15:05,664 - DEBUG - Extracted vm list ['efa300aa-9dfd-4fef-8ce9-996faabf9f65']
2025-09-18 04:15:05,664 - INFO - VM efa300aa-9dfd-4fef-8ce9-996faabf9f65 is present in default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393
2025-09-18 04:15:05,664 - DEBUG - Verifying vm in vrouter uve
2025-09-18 04:15:05,664 - DEBUG - Getting info from collector 10.0.0.41..
2025-09-18 04:15:05,664 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65?flat
2025-09-18 04:15:05,683 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4130-1']
2025-09-18 04:15:05,683 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:15:05,754 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:15:05,849 - DEBUG - VM efa300aa-9dfd-4fef-8ce9-996faabf9f65 is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:15:05,849 - DEBUG - Validated that VM ctest-vn1_vm2-10235123 is in Vrouter an-jenkins-deploy-platform-ansible-os-4130-1 UVE
2025-09-18 04:15:05,849 - DEBUG - Verifying vm tap interface/vn in vrouter uve
2025-09-18 04:15:05,858 - DEBUG - Expected tap interface of VM uuid efa300aa-9dfd-4fef-8ce9-996faabf9f65 is default-domain:ctest-TestBasicVMVN0-22347485:20405166-5e1b-4ebf-8642-8166a1e97b4b
2025-09-18 04:15:05,858 - DEBUG - Expected VN of VM uuid efa300aa-9dfd-4fef-8ce9-996faabf9f65 is default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393
2025-09-18 04:15:05,858 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:15:05,924 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:15:05,956 - DEBUG - VM efa300aa-9dfd-4fef-8ce9-996faabf9f65 is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:15:05,956 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-22347485:20405166-5e1b-4ebf-8642-8166a1e97b4b of vm efa300aa-9dfd-4fef-8ce9-996faabf9f65 is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:15:05,956 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 of vm efa300aa-9dfd-4fef-8ce9-996faabf9f65 is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:15:05,956 - DEBUG - Verifying the efa300aa-9dfd-4fef-8ce9-996faabf9f65 virtual network link through opserver 10.0.0.56
2025-09-18 04:15:05,957 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines
2025-09-18 04:15:05,973 - DEBUG - VM link and name as {'name': 'efa300aa-9dfd-4fef-8ce9-996faabf9f65', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65?flat'}
2025-09-18 04:15:05,973 - DEBUG - Verifying the efa300aa-9dfd-4fef-8ce9-996faabf9f65 virtual network link through opserver 10.0.0.22
2025-09-18 04:15:05,973 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-09-18 04:15:06,054 - DEBUG - VM link and name as {'name': 'efa300aa-9dfd-4fef-8ce9-996faabf9f65', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65?flat'}
2025-09-18 04:15:06,054 - DEBUG - Verifying the efa300aa-9dfd-4fef-8ce9-996faabf9f65 virtual network link through opserver 10.0.0.41
2025-09-18 04:15:06,054 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines
2025-09-18 04:15:06,066 - DEBUG - VM link and name as {'name': 'efa300aa-9dfd-4fef-8ce9-996faabf9f65', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65?flat'}
2025-09-18 04:15:06,066 - INFO - VM ctest-vn1_vm2-10235123 validations in Opserver passed
2025-09-18 04:15:07,495 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-4130-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-4130-2)
2025-09-18 04:15:07,554 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2025-09-18 04:15:12,657 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2025-09-18 04:15:17,762 - DEBUG - VM is still in BUILD state, Expected: ACTIVE
2025-09-18 04:15:22,879 - DEBUG - VM is in ACTIVE state now
2025-09-18 04:15:22,879 - INFO - VM name : ctest-vn2_vm2-62359951
2025-09-18 04:15:23,007 - DEBUG - VM ctest-vn2_vm2-62359951 ID is aa0589b4-e453-4ba8-b9a8-c4e93ec92bda
2025-09-18 04:15:23,107 - DEBUG - VM is in ACTIVE state now
2025-09-18 04:15:23,107 - DEBUG - VM ctest-vn2_vm2-62359951 launched on Node an-jenkins-deploy-platform-ansible-os-4130-2
2025-09-18 04:15:23,213 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda
2025-09-18 04:15:23,225 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda
2025-09-18 04:15:23,236 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/0aba7ca4-11ce-4dae-836a-d3f9a35d1c43
2025-09-18 04:15:23,249 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda
2025-09-18 04:15:23,257 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/0aba7ca4-11ce-4dae-836a-d3f9a35d1c43
2025-09-18 04:15:23,269 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/906a53f7-c4ff-4aeb-9a78-ca818184f27a
2025-09-18 04:15:23,281 - DEBUG - Verifying in api server 10.0.0.41
2025-09-18 04:15:23,281 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda
2025-09-18 04:15:23,290 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/0aba7ca4-11ce-4dae-836a-d3f9a35d1c43
2025-09-18 04:15:23,302 - DEBUG - Requesting: http://10.0.0.41:8082/instance-ip/906a53f7-c4ff-4aeb-9a78-ca818184f27a
2025-09-18 04:15:23,311 - INFO - VM ctest-vn2_vm2-62359951 verfication in all API Servers passed
2025-09-18 04:15:23,338 - DEBUG - VM ctest-vn2_vm2-62359951 Tap interface: {'index': '4', 'name': 'tap0aba7ca4-11', 'uuid': '0aba7ca4-11ce-4dae-836a-d3f9a35d1c43', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937:ctest-vn2-33833937', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937', 'vm_uuid': 'aa0589b4-e453-4ba8-b9a8-c4e93ec92bda', 'vm_name': 'ctest-vn2_vm2-62359951', 'ip_addr': '192.168.2.251', 'mac_addr': '02:0a:ba:7c:a4:11', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '129', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-22347485:0aba7ca4-11ce-4dae-836a-d3f9a35d1c43', 'sg_uuid_list': ['f51220a7-8281-4ea3-8440-e34dbd8dc48d'], 'static_route_list': None, 'vm_project_uuid': '0a1e7454-93bc-4746-b98c-6e3a6282a39d', 'admin_state': 'Enabled', 'flow_key_idx': '34', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-09-18 04:15:23,338 - DEBUG - Agent 10.0.0.27 vrf name: default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937:ctest-vn2-33833937
2025-09-18 04:15:23,344 - DEBUG - Agent VRF Object : {'vrf_list': [{'name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937:ctest-vn2-33833937', 'ucindex': '2', 'mcindex': '2', 'l2index': '2', 'source': 'Config; ', 'uc6index': '2', 'vn': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937', 'table_label': '-1', 'vxlan_id': '13', 'evpnindex': '2', 'brindex': '2', 'mplsindex': '0', 'RD': '10.20.0.192:2', 'mac_aging_time': '0', 'layer2_control_word': 'false', 'forwarding_vrf': None, 'hbf_rintf': '-1', 'hbf_lintf': '-1'}]}
2025-09-18 04:15:23,453 - DEBUG - VM ctest-vn2_vm2-62359951 : Tap interface tap0aba7ca4-11 is set to Active
2025-09-18 04:15:23,453 - DEBUG - Tap interface tap0aba7ca4-11 detail : {'index': '4', 'name': 'tap0aba7ca4-11', 'uuid': '0aba7ca4-11ce-4dae-836a-d3f9a35d1c43', 'vrf_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937:ctest-vn2-33833937', 'active': 'Active', 'ipv4_active': 'Active', 'l2_active': 'L2 Active', 'ip6_active': 'Ipv6 Inactive < no-ipv6-addr >', 'health_check_active': 'Active', 'dhcp_service': 'Enable', 'dhcp_service_v6': 'Disable', 'dns_service': 'Enable', 'type': 'vport', 'label': '30', 'l2_label': '34', 'vxlan_id': '13', 'vn_name': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937', 'vm_uuid': 'aa0589b4-e453-4ba8-b9a8-c4e93ec92bda', 'vm_name': 'ctest-vn2_vm2-62359951', 'ip_addr': '192.168.2.251', 'mac_addr': '02:0a:ba:7c:a4:11', 'policy': 'Enable', 'fip_list': [], 'mdata_ip_addr': '169.254.0.4', 'service_vlan_list': None, 'os_ifindex': '129', 'fabric_port': 'NotFabricPort', 'alloc_linklocal_ip': 'LL-Enable', 'analyzer_name': None, 'config_name': 'default-domain:ctest-TestBasicVMVN0-22347485:0aba7ca4-11ce-4dae-836a-d3f9a35d1c43', 'sg_uuid_list': ['f51220a7-8281-4ea3-8440-e34dbd8dc48d'], 'static_route_list': None, 'vm_project_uuid': '0a1e7454-93bc-4746-b98c-6e3a6282a39d', 'admin_state': 'Enabled', 'flow_key_idx': '34', 'allowed_address_pair_list': None, 'ip6_addr': '::', 'local_preference': '0', 'tx_vlan_id': '-1', 'rx_vlan_id': '-1', 'parent_interface': None, 'subnet': '--NA--', 'sub_type': 'Tap', 'vrf_assign_acl_uuid': '--NA--', 'vmi_type': 'Virtual Machine', 'transport': 'Ethernet', 'logical_interface_uuid': '00000000-0000-0000-0000-000000000000', 'flood_unknown_unicast': 'false', 'physical_device': None, 'physical_interface': None, 'fixed_ip4_list': None, 'fixed_ip6_list': None, 'fat_flow_list': None, 'metadata_ip_active': 'Active', 'service_health_check_ip': '0.0.0.0', 'alias_ip_list': None, 'drop_new_flows': 'false', 'bridge_domain_list': [], 'vmi_tag_list': None, 'policy_set_acl_list': None, 'slo_list': None, 'vhostuser_mode': '0', 'si_other_end_vmi': '00000000-0000-0000-0000-000000000000', 'cfg_igmp_enable': 'false', 'igmp_enabled': 'false', 'max_flows': '0', 'policy_set_fwaas_list': None, 'bond_interface_list': None, 'mac_ip_learning_enable': 'false', 'mac_ip_list': []}
2025-09-18 04:15:23,572 - DEBUG - Starting Layer 2 verification in Agent
2025-09-18 04:15:23,604 - DEBUG - Layer 2 path is seen for VM MAC 02:0a:ba:7c:a4:11 in agent 10.0.0.27
2025-09-18 04:15:23,605 - DEBUG - Active layer 2 route in agent is present for VMI tap0aba7ca4-11
2025-09-18 04:15:23,605 - DEBUG - L2 label(34) matches bw route table and interface table
2025-09-18 04:15:23,634 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 is found in Agent of node 10.0.0.27
2025-09-18 04:15:23,756 - DEBUG - VRF IDs of VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 is consistent in agent 10.0.0.27
2025-09-18 04:15:23,757 - DEBUG - Route for VM IP ['192.168.2.251'] is consistent in agent 10.0.0.27
2025-09-18 04:15:23,757 - DEBUG - VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 verification for VM ctest-vn2_vm2-62359951 in Agent 10.0.0.27 passed
2025-09-18 04:15:23,879 - DEBUG - Starting all layer 2 verification in agent 10.0.0.27
2025-09-18 04:15:23,898 - DEBUG - Route for VM MAC 02:0a:ba:7c:a4:11 is consistent in agent 10.0.0.27
2025-09-18 04:15:27,119 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1026ms')
2025-09-18 04:15:27,119 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-62359951 failed!
2025-09-18 04:15:27,134 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 is 192.168.2.254 and allocation pool is NOT set
2025-09-18 04:15:31,211 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1013ms')
2025-09-18 04:15:31,211 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-62359951 failed!
2025-09-18 04:15:31,228 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 is 192.168.2.254 and allocation pool is NOT set
2025-09-18 04:15:35,314 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms')
2025-09-18 04:15:35,314 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-62359951 failed!
2025-09-18 04:15:35,331 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 is 192.168.2.254 and allocation pool is NOT set
2025-09-18 04:15:39,403 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1003ms')
2025-09-18 04:15:39,404 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-62359951 failed!
2025-09-18 04:15:39,418 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 is 192.168.2.254 and allocation pool is NOT set
2025-09-18 04:15:43,502 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms')
2025-09-18 04:15:43,503 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-62359951 failed!
2025-09-18 04:15:43,518 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 is 192.168.2.254 and allocation pool is NOT set
2025-09-18 04:15:47,604 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1016ms')
2025-09-18 04:15:47,604 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-62359951 failed!
2025-09-18 04:15:47,621 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 is 192.168.2.254 and allocation pool is NOT set
2025-09-18 04:15:51,688 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1001ms')
2025-09-18 04:15:51,688 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-62359951 failed!
2025-09-18 04:15:51,704 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 is 192.168.2.254 and allocation pool is NOT set
2025-09-18 04:15:55,791 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1019ms')
2025-09-18 04:15:55,791 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-62359951 failed!
2025-09-18 04:15:55,806 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 is 192.168.2.254 and allocation pool is NOT set
2025-09-18 04:15:59,886 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1010ms')
2025-09-18 04:15:59,886 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-62359951 failed!
2025-09-18 04:15:59,903 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 is 192.168.2.254 and allocation pool is NOT set
2025-09-18 04:16:03,979 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1014ms')
2025-09-18 04:16:03,979 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-62359951 failed!
2025-09-18 04:16:03,998 - DEBUG - Gateway for vn default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 is 192.168.2.254 and allocation pool is NOT set
2025-09-18 04:16:06,069 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=6.90 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=2.59 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 2.586/4.743/6.900/2.157 ms')
2025-09-18 04:16:06,070 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-62359951 passed
2025-09-18 04:16:06,070 - INFO - VM ctest-vn2_vm2-62359951 verifications in Compute nodes passed
2025-09-18 04:16:06,300 - DEBUG - Validated VM route 192.168.2.251 in vrouter of 10.0.0.27
2025-09-18 04:16:06,308 - DEBUG - Remote Nh validation passed. Expected Tunnel, label: None, 0; Got None,0
2025-09-18 04:16:06,308 - INFO - Validated routes of VM ctest-vn2_vm2-62359951 in all vrouters
2025-09-18 04:16:06,733 - DEBUG - Starting all layer2 verification in 10.0.0.22 Control Node
2025-09-18 04:16:06,860 - DEBUG - Layer2 route found for VM MAC 02:0a:ba:7c:a4:11 in Control-node 10.0.0.22
2025-09-18 04:16:06,860 - DEBUG - L2 Label for VM ctest-vn2_vm2-62359951 same between Control-node 10.0.0.22 and Agent, Expected: 34, Seen: 34
2025-09-18 04:16:06,860 - DEBUG - Starting all layer2 verification in 10.0.0.41 Control Node
2025-09-18 04:16:06,988 - DEBUG - Layer2 route found for VM MAC 02:0a:ba:7c:a4:11 in Control-node 10.0.0.41
2025-09-18 04:16:06,988 - DEBUG - L2 Label for VM ctest-vn2_vm2-62359951 same between Control-node 10.0.0.41 and Agent, Expected: 34, Seen: 34
2025-09-18 04:16:06,988 - DEBUG - Starting all layer2 verification in 10.0.0.56 Control Node
2025-09-18 04:16:07,118 - DEBUG - Layer2 route found for VM MAC 02:0a:ba:7c:a4:11 in Control-node 10.0.0.56
2025-09-18 04:16:07,118 - DEBUG - L2 Label for VM ctest-vn2_vm2-62359951 same between Control-node 10.0.0.56 and Agent, Expected: 34, Seen: 34
2025-09-18 04:16:07,118 - INFO - Verification in Control-nodes for VM ctest-vn2_vm2-62359951 passed
2025-09-18 04:16:07,118 - DEBUG - Verifying the vm in opserver
2025-09-18 04:16:07,118 - DEBUG - Verifying in collector 10.0.0.41 ...
2025-09-18 04:16:07,118 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda?flat
2025-09-18 04:16:07,136 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-22347485:0aba7ca4-11ce-4dae-836a-d3f9a35d1c43?flat
2025-09-18 04:16:07,158 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-22347485:0aba7ca4-11ce-4dae-836a-d3f9a35d1c43?flat
2025-09-18 04:16:07,180 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda?flat
2025-09-18 04:16:07,196 - DEBUG - Verifying vm in vn uve
2025-09-18 04:16:07,196 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machine-interface/default-domain:ctest-TestBasicVMVN0-22347485:0aba7ca4-11ce-4dae-836a-d3f9a35d1c43?flat
2025-09-18 04:16:07,221 - DEBUG - VM uve shows interface as {'admin_state': True, 'is_health_check_active': True, 'fixed_ip4_list': ['192.168.2.251'], 'ip6_address': '::', 'tx_vlan': 65535, 'virtual_network': 'default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937', 'ip6_active': False, 'port_mirror_enabled': False, 'uuid': '0aba7ca4-11ce-4dae-836a-d3f9a35d1c43', 'ip4_active': True, 'vm_name': 'ctest-vn2_vm2-62359951', 'vm_uuid': 'aa0589b4-e453-4ba8-b9a8-c4e93ec92bda', 'l2_active': True, 'mac_address': '02:0a:ba:7c:a4:11', 'ip_address': '192.168.2.251', 'rx_vlan': 65535, 'vn_uuid': '81b38e96-1391-45f9-9c28-5b67dd101541', 'active': True, '__T': 1758168935631036, 'vhostuser_mode': 0, 'gateway': '192.168.2.254', 'label': 30}
2025-09-18 04:16:07,221 - DEBUG - VM uve shows ip address as ['192.168.2.251', '::']
2025-09-18 04:16:07,221 - DEBUG - VM uve shows virtual network as default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937
2025-09-18 04:16:07,221 - INFO - Verifying through opserver in 10.0.0.56
2025-09-18 04:16:07,221 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937?flat
2025-09-18 04:16:07,246 - DEBUG - expected vm list aa0589b4-e453-4ba8-b9a8-c4e93ec92bda
2025-09-18 04:16:07,246 - DEBUG - Extracted vm list ['aa0589b4-e453-4ba8-b9a8-c4e93ec92bda']
2025-09-18 04:16:07,246 - INFO - VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is present in default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937
2025-09-18 04:16:07,246 - DEBUG - Verifying vm in vrouter uve
2025-09-18 04:16:07,247 - DEBUG - Getting info from collector 10.0.0.41..
2025-09-18 04:16:07,247 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda?flat
2025-09-18 04:16:07,262 - DEBUG - VM uve shows vrouter as ['an-jenkins-deploy-platform-ansible-os-4130-2']
2025-09-18 04:16:07,263 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:16:07,346 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:16:07,377 - DEBUG - VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:16:07,377 - DEBUG - Validated that VM ctest-vn2_vm2-62359951 is in Vrouter an-jenkins-deploy-platform-ansible-os-4130-2 UVE
2025-09-18 04:16:07,377 - DEBUG - Verifying vm tap interface/vn in vrouter uve
2025-09-18 04:16:07,395 - DEBUG - Expected tap interface of VM uuid aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is default-domain:ctest-TestBasicVMVN0-22347485:0aba7ca4-11ce-4dae-836a-d3f9a35d1c43
2025-09-18 04:16:07,395 - DEBUG - Expected VN of VM uuid aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937
2025-09-18 04:16:07,395 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:16:07,462 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:16:07,494 - DEBUG - VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:16:07,494 - INFO - tap interface default-domain:ctest-TestBasicVMVN0-22347485:0aba7ca4-11ce-4dae-836a-d3f9a35d1c43 of vm aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:16:07,494 - DEBUG - Connected network default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 of vm aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:16:07,494 - DEBUG - Verifying the aa0589b4-e453-4ba8-b9a8-c4e93ec92bda virtual network link through opserver 10.0.0.56
2025-09-18 04:16:07,495 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines
2025-09-18 04:16:07,511 - DEBUG - VM link and name as {'name': 'aa0589b4-e453-4ba8-b9a8-c4e93ec92bda', 'href': 'http://10.0.0.56:8081/analytics/uves/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda?flat'}
2025-09-18 04:16:07,511 - DEBUG - Verifying the aa0589b4-e453-4ba8-b9a8-c4e93ec92bda virtual network link through opserver 10.0.0.22
2025-09-18 04:16:07,511 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-09-18 04:16:07,525 - DEBUG - VM link and name as {'name': 'aa0589b4-e453-4ba8-b9a8-c4e93ec92bda', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda?flat'}
2025-09-18 04:16:07,525 - DEBUG - Verifying the aa0589b4-e453-4ba8-b9a8-c4e93ec92bda virtual network link through opserver 10.0.0.41
2025-09-18 04:16:07,525 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines
2025-09-18 04:16:07,538 - DEBUG - VM link and name as {'name': 'aa0589b4-e453-4ba8-b9a8-c4e93ec92bda', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda?flat'}
2025-09-18 04:16:07,539 - INFO - VM ctest-vn2_vm2-62359951 validations in Opserver passed
2025-09-18 04:16:07,539 - INFO - Waiting for VM ctest-vn1_vm2-10235123 to be up..
2025-09-18 04:16:07,539 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:16:07,539 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65
2025-09-18 04:16:07,549 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/20405166-5e1b-4ebf-8642-8166a1e97b4b
2025-09-18 04:16:08,722 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.85 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.550 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.550/2.198/3.847/1.648 ms')
2025-09-18 04:16:08,722 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-10235123 passed
2025-09-18 04:16:08,800 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:16:08,801 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-10235123, IP 192.168.1.251, Port 22
2025-09-18 04:16:08,972 - DEBUG - VM ctest-vn1_vm2-10235123 is ready for SSH connections
2025-09-18 04:16:08,972 - INFO - Waiting for VM ctest-vn2_vm2-62359951 to be up..
2025-09-18 04:16:08,972 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:16:08,972 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda
2025-09-18 04:16:08,982 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine-interface/0aba7ca4-11ce-4dae-836a-d3f9a35d1c43
2025-09-18 04:16:10,150 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=3.76 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.574 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.574/2.164/3.755/1.590 ms')
2025-09-18 04:16:10,150 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn2_vm2-62359951 passed
2025-09-18 04:16:10,227 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:16:10,227 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-62359951, IP 192.168.2.251, Port 22
2025-09-18 04:16:10,295 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-09-18 04:16:10,397 - DEBUG - VM ctest-vn2_vm2-62359951 is NOT ready for SSH connections, VM status: ACTIVE
2025-09-18 04:16:15,399 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:16:15,399 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-62359951, IP 192.168.2.251, Port 22
2025-09-18 04:16:15,467 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-09-18 04:16:15,554 - DEBUG - VM ctest-vn2_vm2-62359951 is NOT ready for SSH connections, VM status: ACTIVE
2025-09-18 04:16:20,554 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:16:20,554 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-62359951, IP 192.168.2.251, Port 22
2025-09-18 04:16:20,625 - DEBUG - Error on ssh to ubuntu@169.254.0.4:22, result: /bin/bash: connect: Connection refused
/bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''}
2025-09-18 04:16:20,737 - DEBUG - VM ctest-vn2_vm2-62359951 is NOT ready for SSH connections, VM status: ACTIVE
2025-09-18 04:16:25,738 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:16:25,738 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-62359951, IP 192.168.2.251, Port 22
2025-09-18 04:16:25,916 - DEBUG - VM ctest-vn2_vm2-62359951 is ready for SSH connections
2025-09-18 04:16:25,916 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set
2025-09-18 04:16:25,916 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 192.168.2.251, host_string: ubuntu@169.254.0.4, password: ubuntugateway: ubuntu@10.0.0.61, gateway password: c0ntrail123
2025-09-18 04:16:25,916 - DEBUG - ping -s 56 -c 3 -W 1 192.168.2.251
2025-09-18 04:17:08,717 - DEBUG - PING 192.168.2.251 (192.168.2.251) 56(84) bytes of data.
64 bytes from 192.168.2.251: icmp_req=1 ttl=63 time=7.57 ms
64 bytes from 192.168.2.251: icmp_req=2 ttl=63 time=1.41 ms
64 bytes from 192.168.2.251: icmp_req=3 ttl=63 time=1.83 ms
--- 192.168.2.251 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2005ms
rtt min/avg/max/mdev = 1.416/3.610/7.576/2.809 ms
2025-09-18 04:17:08,717 - INFO - Ping to IP 192.168.2.251 from VM ctest-vn1_vm2-10235123 passed
2025-09-18 04:17:08,718 - INFO - Deleting VM ctest-vn2_vm2-62359951
2025-09-18 04:17:08,817 - DEBUG - Verifying in api server 10.0.0.41
2025-09-18 04:17:08,817 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda
2025-09-18 04:17:08,828 - DEBUG - VM ID aa0589b4-e453-4ba8-b9a8-c4e93ec92bda of VM ctest-vn2_vm2-62359951 is still found in API Server
2025-09-18 04:17:10,828 - DEBUG - Verifying in api server 10.0.0.41
2025-09-18 04:17:10,829 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda
2025-09-18 04:17:10,837 - DEBUG - Response Code: 404
2025-09-18 04:17:10,837 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda
2025-09-18 04:17:10,897 - DEBUG - Response Code: 404
2025-09-18 04:17:10,898 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/aa0589b4-e453-4ba8-b9a8-c4e93ec92bda
2025-09-18 04:17:10,905 - DEBUG - Response Code: 404
2025-09-18 04:17:10,905 - INFO - VM ctest-vn2_vm2-62359951 is fully removed in API-Server
2025-09-18 04:17:11,068 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found
2025-09-18 04:17:11,109 - DEBUG - Validated that vrouter 10.0.0.61 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937
2025-09-18 04:17:11,279 - DEBUG - VM route 192.168.2.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 of compute 10.0.0.27
2025-09-18 04:17:11,286 - DEBUG - Validated that vrouter 10.0.0.27 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937
2025-09-18 04:17:11,287 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937
2025-09-18 04:17:11,287 - INFO - VM ctest-vn2_vm2-62359951 is removed in Compute, and routes are removed in all compute nodes
2025-09-18 04:17:11,530 - INFO - Routes for VM ctest-vn2_vm2-62359951 is removed in all control-nodes
2025-09-18 04:17:11,530 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:11,599 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:17:11,633 - DEBUG - VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:17:11,633 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:11,634 - DEBUG - interface for vm aa0589b4-e453-4ba8-b9a8-c4e93ec92bda deleted from vrouter uve ...
2025-09-18 04:17:11,634 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:14,635 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:14,700 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:17:14,732 - DEBUG - VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:17:14,732 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:14,732 - DEBUG - interface for vm aa0589b4-e453-4ba8-b9a8-c4e93ec92bda deleted from vrouter uve ...
2025-09-18 04:17:14,732 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:17,733 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:17,811 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:17:17,846 - DEBUG - VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:17:17,846 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:17,846 - DEBUG - interface for vm aa0589b4-e453-4ba8-b9a8-c4e93ec92bda deleted from vrouter uve ...
2025-09-18 04:17:17,846 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:20,847 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:20,915 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:17:20,948 - DEBUG - VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:17:20,948 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:20,948 - DEBUG - interface for vm aa0589b4-e453-4ba8-b9a8-c4e93ec92bda deleted from vrouter uve ...
2025-09-18 04:17:20,948 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:23,948 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:24,029 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:17:24,059 - DEBUG - VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:17:24,059 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:24,059 - DEBUG - interface for vm aa0589b4-e453-4ba8-b9a8-c4e93ec92bda deleted from vrouter uve ...
2025-09-18 04:17:24,060 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:27,060 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:27,127 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:17:27,165 - DEBUG - VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:17:27,165 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:27,165 - DEBUG - interface for vm aa0589b4-e453-4ba8-b9a8-c4e93ec92bda deleted from vrouter uve ...
2025-09-18 04:17:27,165 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:30,166 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:30,238 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:17:30,273 - DEBUG - VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:17:30,273 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:30,273 - DEBUG - interface for vm aa0589b4-e453-4ba8-b9a8-c4e93ec92bda deleted from vrouter uve ...
2025-09-18 04:17:30,273 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:33,274 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:33,341 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:17:33,373 - DEBUG - VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:17:33,373 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:33,373 - DEBUG - interface for vm aa0589b4-e453-4ba8-b9a8-c4e93ec92bda deleted from vrouter uve ...
2025-09-18 04:17:33,373 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:36,373 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-2:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:36,443 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-2?flat
2025-09-18 04:17:36,481 - DEBUG - VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is not present in vrouter an-jenkins-deploy-platform-ansible-os-4130-2 uve
2025-09-18 04:17:36,481 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:36,481 - DEBUG - interface for vm aa0589b4-e453-4ba8-b9a8-c4e93ec92bda deleted from vrouter uve ...
2025-09-18 04:17:36,481 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:36,481 - DEBUG - Validated that VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is removed in Vrouter UVE
2025-09-18 04:17:36,481 - DEBUG - Verifying through opserver in 10.0.0.56
2025-09-18 04:17:36,482 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937?flat
2025-09-18 04:17:36,511 - DEBUG - aa0589b4-e453-4ba8-b9a8-c4e93ec92bda BM not in default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn2-33833937 uve
2025-09-18 04:17:36,511 - DEBUG - Validated that VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is not present in the VN UVE
2025-09-18 04:17:36,511 - DEBUG - Verifying the aa0589b4-e453-4ba8-b9a8-c4e93ec92bda virtual network link through opserver 10.0.0.56
2025-09-18 04:17:36,511 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines
2025-09-18 04:17:36,528 - INFO - aa0589b4-e453-4ba8-b9a8-c4e93ec92bda vm uve deleted from opserver
2025-09-18 04:17:36,528 - DEBUG - Verifying the aa0589b4-e453-4ba8-b9a8-c4e93ec92bda virtual network link through opserver 10.0.0.22
2025-09-18 04:17:36,528 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-09-18 04:17:36,543 - INFO - aa0589b4-e453-4ba8-b9a8-c4e93ec92bda vm uve deleted from opserver
2025-09-18 04:17:36,543 - DEBUG - Verifying the aa0589b4-e453-4ba8-b9a8-c4e93ec92bda virtual network link through opserver 10.0.0.41
2025-09-18 04:17:36,543 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines
2025-09-18 04:17:36,556 - INFO - aa0589b4-e453-4ba8-b9a8-c4e93ec92bda vm uve deleted from opserver
2025-09-18 04:17:36,556 - INFO - Validated that VM aa0589b4-e453-4ba8-b9a8-c4e93ec92bda is removed from Opserver
2025-09-18 04:17:36,556 - INFO - Deleting VM ctest-vn1_vm2-10235123
2025-09-18 04:17:36,643 - DEBUG - Verifying in api server 10.0.0.41
2025-09-18 04:17:36,643 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65
2025-09-18 04:17:36,653 - DEBUG - VM ID efa300aa-9dfd-4fef-8ce9-996faabf9f65 of VM ctest-vn1_vm2-10235123 is still found in API Server
2025-09-18 04:17:38,654 - DEBUG - Verifying in api server 10.0.0.41
2025-09-18 04:17:38,654 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65
2025-09-18 04:17:38,663 - DEBUG - Response Code: 404
2025-09-18 04:17:38,663 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65
2025-09-18 04:17:38,670 - DEBUG - Response Code: 404
2025-09-18 04:17:38,670 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-machine/efa300aa-9dfd-4fef-8ce9-996faabf9f65
2025-09-18 04:17:38,677 - DEBUG - Response Code: 404
2025-09-18 04:17:38,677 - INFO - VM ctest-vn1_vm2-10235123 is fully removed in API-Server
2025-09-18 04:17:38,841 - DEBUG - Skipping VM-deletion-check in nova db since mysql_token is not found
2025-09-18 04:17:39,038 - DEBUG - VM route 192.168.1.251 has been marked for discard in VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 of compute 10.0.0.61
2025-09-18 04:17:39,045 - DEBUG - Validated that vrouter 10.0.0.61 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393
2025-09-18 04:17:39,045 - DEBUG - Validated that vrouter 10.0.0.27 does not have VMs route for VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393
2025-09-18 04:17:39,045 - INFO - Validated that all vrouters do not have VMs route for VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393
2025-09-18 04:17:39,045 - INFO - VM ctest-vn1_vm2-10235123 is removed in Compute, and routes are removed in all compute nodes
2025-09-18 04:17:39,269 - INFO - Routes for VM ctest-vn1_vm2-10235123 is removed in all control-nodes
2025-09-18 04:17:39,269 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:39,377 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:17:39,415 - DEBUG - VM efa300aa-9dfd-4fef-8ce9-996faabf9f65 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:17:39,415 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:39,415 - DEBUG - interface for vm efa300aa-9dfd-4fef-8ce9-996faabf9f65 deleted from vrouter uve ...
2025-09-18 04:17:39,415 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:42,416 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:42,480 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:17:42,514 - DEBUG - VM efa300aa-9dfd-4fef-8ce9-996faabf9f65 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:17:42,515 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:42,515 - DEBUG - interface for vm efa300aa-9dfd-4fef-8ce9-996faabf9f65 deleted from vrouter uve ...
2025-09-18 04:17:42,515 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:45,515 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:45,580 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:17:45,611 - DEBUG - VM efa300aa-9dfd-4fef-8ce9-996faabf9f65 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:17:45,611 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:45,611 - DEBUG - interface for vm efa300aa-9dfd-4fef-8ce9-996faabf9f65 deleted from vrouter uve ...
2025-09-18 04:17:45,611 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:48,611 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:48,681 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:17:48,712 - DEBUG - VM efa300aa-9dfd-4fef-8ce9-996faabf9f65 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:17:48,713 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:48,713 - DEBUG - interface for vm efa300aa-9dfd-4fef-8ce9-996faabf9f65 deleted from vrouter uve ...
2025-09-18 04:17:48,713 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:51,713 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:51,778 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:17:51,812 - DEBUG - VM efa300aa-9dfd-4fef-8ce9-996faabf9f65 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:17:51,812 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:51,812 - DEBUG - interface for vm efa300aa-9dfd-4fef-8ce9-996faabf9f65 deleted from vrouter uve ...
2025-09-18 04:17:51,812 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:54,813 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:54,878 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:17:54,910 - DEBUG - VM efa300aa-9dfd-4fef-8ce9-996faabf9f65 is still present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:17:54,910 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:54,910 - DEBUG - interface for vm efa300aa-9dfd-4fef-8ce9-996faabf9f65 deleted from vrouter uve ...
2025-09-18 04:17:54,910 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:57,911 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-4130-1:Compute:contrail-vrouter-agent:0?flat
2025-09-18 04:17:57,978 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-4130-1?flat
2025-09-18 04:17:58,010 - DEBUG - VM efa300aa-9dfd-4fef-8ce9-996faabf9f65 is not present in vrouter an-jenkins-deploy-platform-ansible-os-4130-1 uve
2025-09-18 04:17:58,010 - DEBUG - Verifying if the vm interface deleted from vroter uve...
2025-09-18 04:17:58,010 - DEBUG - interface for vm efa300aa-9dfd-4fef-8ce9-996faabf9f65 deleted from vrouter uve ...
2025-09-18 04:17:58,010 - DEBUG - Deleted interface not in error interface list ...
2025-09-18 04:17:58,010 - DEBUG - Validated that VM efa300aa-9dfd-4fef-8ce9-996faabf9f65 is removed in Vrouter UVE
2025-09-18 04:17:58,010 - DEBUG - Verifying through opserver in 10.0.0.56
2025-09-18 04:17:58,010 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-network/default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393?flat
2025-09-18 04:17:58,037 - DEBUG - efa300aa-9dfd-4fef-8ce9-996faabf9f65 BM not in default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393 uve
2025-09-18 04:17:58,037 - DEBUG - Validated that VM efa300aa-9dfd-4fef-8ce9-996faabf9f65 is not present in the VN UVE
2025-09-18 04:17:58,037 - DEBUG - Verifying the efa300aa-9dfd-4fef-8ce9-996faabf9f65 virtual network link through opserver 10.0.0.56
2025-09-18 04:17:58,037 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/virtual-machines
2025-09-18 04:17:58,051 - INFO - efa300aa-9dfd-4fef-8ce9-996faabf9f65 vm uve deleted from opserver
2025-09-18 04:17:58,051 - DEBUG - Verifying the efa300aa-9dfd-4fef-8ce9-996faabf9f65 virtual network link through opserver 10.0.0.22
2025-09-18 04:17:58,051 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-machines
2025-09-18 04:17:58,066 - INFO - efa300aa-9dfd-4fef-8ce9-996faabf9f65 vm uve deleted from opserver
2025-09-18 04:17:58,066 - DEBUG - Verifying the efa300aa-9dfd-4fef-8ce9-996faabf9f65 virtual network link through opserver 10.0.0.41
2025-09-18 04:17:58,066 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-machines
2025-09-18 04:17:58,079 - INFO - efa300aa-9dfd-4fef-8ce9-996faabf9f65 vm uve deleted from opserver
2025-09-18 04:17:58,079 - INFO - Validated that VM efa300aa-9dfd-4fef-8ce9-996faabf9f65 is removed from Opserver
2025-09-18 04:17:58,080 - INFO - Deleting VN ctest-vn3-65676010
2025-09-18 04:17:58,231 - INFO - Deleting VM ctest-vn2_vm1-37053484
2025-09-18 04:17:58,347 - INFO - Deleting VM ctest-vn1_vm1-99922935
2025-09-18 04:17:58,606 - INFO - Deleting VN ctest-vn2-33833937
2025-09-18 04:17:58,669 - DEBUG - RefsExistError Delete when resource still referred: ['http://10.0.0.41:8082/virtual-machine-interface/b06272d7-0045-449a-b851-153226aebfc9'] while deleting VN ctest-vn2-33833937..Will retry
2025-09-18 04:18:00,828 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/ebaa7738-37be-48f5-920d-e73acd625479
2025-09-18 04:18:00,834 - DEBUG - Response Code: 404
2025-09-18 04:18:00,835 - DEBUG - Requesting: http://10.0.0.56:8082/domains
2025-09-18 04:18:00,845 - DEBUG - Requesting: http://10.0.0.56:8082/domain/245899c5-19ba-4930-be6a-859f116860e1
2025-09-18 04:18:00,880 - DEBUG - Requesting: http://10.0.0.56:8082/project/0a1e7454-93bc-4746-b98c-6e3a6282a39d
2025-09-18 04:18:01,035 - INFO - Validated that VN ctest-vn2-33833937 is not found in API Server
2025-09-18 04:18:01,049 - DEBUG - VN ctest-vn2-33833937 is not present in Agent 10.0.0.61
2025-09-18 04:18:01,065 - DEBUG - VN ctest-vn2-33833937 is not present in Agent 10.0.0.27
2025-09-18 04:18:01,065 - INFO - Validated that VN ctest-vn2-33833937 is not in any agent
2025-09-18 04:18:01,111 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-33833937 info
2025-09-18 04:18:01,318 - INFO - Deleting VN ctest-vn1-44999393
2025-09-18 04:18:01,472 - DEBUG - Requesting: http://10.0.0.56:8082/routing-instance/47ac12e7-92e7-45d0-8989-7225debe3778
2025-09-18 04:18:01,484 - DEBUG - Response Code: 404
2025-09-18 04:18:01,484 - DEBUG - Requesting: http://10.0.0.56:8082/domains
2025-09-18 04:18:01,499 - DEBUG - Requesting: http://10.0.0.56:8082/domain/245899c5-19ba-4930-be6a-859f116860e1
2025-09-18 04:18:01,529 - DEBUG - Requesting: http://10.0.0.56:8082/project/0a1e7454-93bc-4746-b98c-6e3a6282a39d
2025-09-18 04:18:01,645 - INFO - Validated that VN ctest-vn1-44999393 is not found in API Server
2025-09-18 04:18:01,659 - DEBUG - VN ctest-vn1-44999393 is not present in Agent 10.0.0.61
2025-09-18 04:18:01,673 - DEBUG - VN ctest-vn1-44999393 is not present in Agent 10.0.0.27
2025-09-18 04:18:01,673 - INFO - Validated that VN ctest-vn1-44999393 is not in any agent
2025-09-18 04:18:01,679 - DEBUG - VRF 2 is not seen in agent 10.0.0.61
2025-09-18 04:18:01,686 - DEBUG - Vrouter 10.0.0.61 does not have vrf 2 for VN ctest-vn1-44999393
2025-09-18 04:18:01,686 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-TestBasicVMVN0-22347485:ctest-vn1-44999393
2025-09-18 04:18:01,710 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-44999393 info
2025-09-18 04:18:01,763 - INFO - Deleted policy policy2
2025-09-18 04:18:01,807 - INFO - Deleted policy policy1
2025-09-18 04:18:03,072 - DEBUG - Skipping xmpp flap check
2025-09-18 04:18:03,072 - INFO - END TEST : test_process_restart_in_policy_between_vns : PASSED[0:07:27]
2025-09-18 04:18:03,073 - INFO - --------------------------------------------------------------------------------
2025-09-18 04:18:03,075 - INFO - ================================================================================
2025-09-18 04:18:03,075 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling
2025-09-18 04:18:03,076 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229).
1. Send broadcast traffic from one compute node.
2. Other compute in same subnet should receive that traffic.
3. Receiving compute should treat this traffic as underlay.
4. Compute should not replicate the packet and send the copy back.
Pass criteria: Step 3-4 should pass
Maintainer : chhandak@juniper.net
2025-09-18 04:18:04,375 - DEBUG - Skipping xmpp flap check
2025-09-18 04:18:04,375 - INFO - Initial checks done. Running the testcase now
2025-09-18 04:18:04,375 - INFO -
2025-09-18 04:18:04,781 - INFO - Agent interface name: ens6
2025-09-18 04:18:04,866 - DEBUG - Executing command: sudo tcpdump -nni ens6 -U host 0.0.0.0 -w /tmp/ens6_ctest-random-85884719.pcap
2025-09-18 04:18:10,256 - INFO - Pinging broacast address 0.0.0.0 from compute 10.20.0.129
2025-09-18 04:18:24,556 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}')
2025-09-18 04:18:26,694 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-85884719.pcap | grep -c length
2025-09-18 04:18:26,709 - DEBUG - STDOUT: 2
2025-09-18 04:18:26,709 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-85884719.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2025-09-18 04:18:26,709 - WARNING - 2 packets are found in tcpdump output file /tmp/ens6_ctest-random-85884719.pcap but expected 10
2025-09-18 04:18:28,710 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-85884719.pcap | grep -c length
2025-09-18 04:18:28,731 - DEBUG - STDOUT: 2
2025-09-18 04:18:28,731 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-85884719.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2025-09-18 04:18:28,731 - WARNING - 2 packets are found in tcpdump output file /tmp/ens6_ctest-random-85884719.pcap but expected 10
2025-09-18 04:18:30,732 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-85884719.pcap | grep -c length
2025-09-18 04:18:30,750 - DEBUG - STDOUT: 2
2025-09-18 04:18:30,750 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-85884719.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2025-09-18 04:18:30,750 - WARNING - 2 packets are found in tcpdump output file /tmp/ens6_ctest-random-85884719.pcap but expected 10
2025-09-18 04:18:32,751 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-85884719.pcap | grep -c length
2025-09-18 04:18:32,766 - DEBUG - STDOUT: 2
2025-09-18 04:18:32,766 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-85884719.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2025-09-18 04:18:32,766 - WARNING - 2 packets are found in tcpdump output file /tmp/ens6_ctest-random-85884719.pcap but expected 10
2025-09-18 04:18:34,766 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-85884719.pcap | grep -c length
2025-09-18 04:18:34,785 - DEBUG - STDOUT: 2
2025-09-18 04:18:34,785 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-85884719.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2025-09-18 04:18:34,785 - WARNING - 2 packets are found in tcpdump output file /tmp/ens6_ctest-random-85884719.pcap but expected 10
2025-09-18 04:18:36,786 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-85884719.pcap | grep -c length
2025-09-18 04:18:36,803 - DEBUG - STDOUT: 2
2025-09-18 04:18:36,803 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-85884719.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2025-09-18 04:18:36,803 - WARNING - 2 packets are found in tcpdump output file /tmp/ens6_ctest-random-85884719.pcap but expected 10
2025-09-18 04:18:38,803 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-85884719.pcap | grep -c length
2025-09-18 04:18:38,819 - DEBUG - STDOUT: 2
2025-09-18 04:18:38,819 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-85884719.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2025-09-18 04:18:38,819 - WARNING - 2 packets are found in tcpdump output file /tmp/ens6_ctest-random-85884719.pcap but expected 10
2025-09-18 04:18:40,089 - ERROR - AssertionError
Python 3.9.21: /usr/bin/python3
Thu Sep 18 04:18:38 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={})
78 log.info('Initial checks done. Running the testcase now')
79 log.info('')
80 result = function(self, *args, **kwargs)
81 if self.inputs.upgrade:
82 pid = os.getpid()
result = None
function =
self =
args = ()
kwargs = {}
/contrail-test/serial_scripts/vm_regression/test_vm_serial.py in test_underlay_broadcast_traffic_handling(self=)
948
949 # Analyze pcap
950 assert verify_tcpdump_count(self, session, pcap, exp_count=packet_count), "There should only be %s\
951 packet from source %s on compute %s" %(packet_count, broadcast_address, compute_ip)
952 self.logger.info(
global verify_tcpdump_count = .deco_retry..f_retry>
self =
session =
pcap = '/tmp/ens6_ctest-random-85884719.pcap'
exp_count undefined
packet_count = 10
broadcast_address = '0.0.0.0'
compute_ip = '10.0.0.27'
AssertionError: There should only be 10 packet from source 0.0.0.0 on compute 10.0.0.27
__cause__ = None
__class__ =
__context__ = None
__delattr__ =
__dict__ = {}
__dir__ =
__doc__ = 'Assertion failed.'
__eq__ =
__format__ =
__ge__ =
__getattribute__ =
__gt__ =
__hash__ =
__init__ =
__init_subclass__ =
__le__ =
__lt__ =
__ne__ =
__new__ =
__reduce__ =
__reduce_ex__ =
__repr__ =
__setattr__ =
__setstate__ =
__sizeof__ =
__str__ =
__subclasshook__ =
__suppress_context__ = False
__traceback__ =
args = ('There should only be 10 ... packet from source 0.0.0.0 on compute 10.0.0.27',)
with_traceback =
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
result = function(self, *args, **kwargs)
File "/contrail-test/serial_scripts/vm_regression/test_vm_serial.py", line 950, in test_underlay_broadcast_traffic_handling
assert verify_tcpdump_count(self, session, pcap, exp_count=packet_count), "There should only be %s\
AssertionError: There should only be 10 packet from source 0.0.0.0 on compute 10.0.0.27
2025-09-18 04:18:40,089 - DEBUG - Skipping xmpp flap check
2025-09-18 04:18:40,089 - INFO -
2025-09-18 04:18:40,090 - INFO - END TEST : test_underlay_broadcast_traffic_handling : FAILED[0:00:37]
2025-09-18 04:18:40,090 - INFO - --------------------------------------------------------------------------------
2025-09-18 04:18:40,962 - INFO - Deleted project: ctest-TestBasicVMVN0-22347485, ID : 0a1e7454-93bc-4746-b98c-6e3a6282a39d