2025-08-04 03:56:14,522 - INFO - Domain Default found not creating 2025-08-04 03:56:14,699 - INFO - Project ctest-AnalyticsTestSanityWithResource-33294378 not found, creating it 2025-08-04 03:56:15,311 - INFO - Created Project:ctest-AnalyticsTestSanityWithResource-33294378, ID : a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:56:17,611 - DEBUG - Response for create_network : {'network': {'id': 'f4f2b026-bd63-4f2f-b7d1-2cc78ff80148', 'name': 'ctest-vn1-08779962', 'tenant_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'project_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'ctest-vn1-08779962'], 'port_security_enabled': True, 'description': ''}} 2025-08-04 03:56:17,878 - DEBUG - Response for create_subnet : {'subnet': {'id': '62dad8db-c361-4be0-ba79-889d78e2cc4f', 'name': '', 'tenant_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'network_id': 'f4f2b026-bd63-4f2f-b7d1-2cc78ff80148', 'ip_version': 4, 'cidr': '214.167.83.192/26', 'allocation_pools': [{'start': '214.167.83.194', 'end': '214.167.83.254'}], 'gateway_ip': '214.167.83.193', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '214.167.83.194', 'tags': [], 'project_id': 'a48ab5304ea04d359e730dcdaaab3e63'}} 2025-08-04 03:56:17,906 - INFO - Created VN ctest-vn1-08779962 2025-08-04 03:56:17,924 - DEBUG - VN ctest-vn1-08779962 UUID is f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:56:18,051 - DEBUG - Services list from nova: [, , , ] 2025-08-04 03:56:18,051 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3750-2', 'an-jenkins-deploy-platform-ansible-os-3750-1']} 2025-08-04 03:56:19,220 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3750-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3750-2) 2025-08-04 03:56:20,642 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3750-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3750-2) 2025-08-04 03:56:20,643 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:56:21,051 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:56:21,089 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:56:21,120 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:56:21,245 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:56:21,309 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:56:21,343 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:56:21,370 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:56:21,381 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 03:56:21,389 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:56:21,390 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:56:21,428 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:56:21,458 - INFO - Verified VN network id 13 for VN f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:56:21,458 - INFO - Verifications in API Server for VN ctest-vn1-08779962 passed 2025-08-04 03:56:21,458 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:56:21,472 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:56:21,483 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:56:21,493 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 03:56:21,521 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:56:17', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:03.699748'} 2025-08-04 03:56:21,537 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:56:21,545 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:56:17', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:03.730660'} 2025-08-04 03:56:21,562 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:56:21,570 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:56:17', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:03.754013'} 2025-08-04 03:56:21,581 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:56:21,581 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-08779962 passed 2025-08-04 03:56:21,581 - DEBUG - ====Verifying policy data for ctest-vn1-08779962 in API_Server ====== 2025-08-04 03:56:21,581 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:56:21,603 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:56:21,651 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:56:21,751 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:56:21,770 - DEBUG - =>VN ctest-vn1-08779962 has no policy to be verified 2025-08-04 03:56:21,770 - DEBUG - Verifying the vn in opserver 2025-08-04 03:56:21,770 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.27 2025-08-04 03:56:21,770 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:56:22,095 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:56:22,172 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:56:22,172 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.38 2025-08-04 03:56:22,172 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:56:22,481 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:56:22,547 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:56:22,547 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.61 2025-08-04 03:56:22,547 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:56:22,854 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:56:22,922 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:56:22,922 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 is found in opserver 2025-08-04 03:56:22,937 - DEBUG - VRF ids for VN ctest-vn1-08779962: {'10.0.0.57': '2'} 2025-08-04 03:56:22,937 - INFO - Waiting for VM ctest-vn1_vm1-33302525 to be up.. 2025-08-04 03:56:23,027 - DEBUG - VM is in ACTIVE state now 2025-08-04 03:56:23,028 - INFO - VM name : ctest-vn1_vm1-33302525 2025-08-04 03:56:23,115 - DEBUG - VM ctest-vn1_vm1-33302525 ID is acfc709a-44b1-4bca-87d2-f365690f25b6 2025-08-04 03:56:23,140 - DEBUG - VM ctest-vn1_vm1-33302525 launched on Node an-jenkins-deploy-platform-ansible-os-3750-2 2025-08-04 03:56:23,229 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/acfc709a-44b1-4bca-87d2-f365690f25b6 2025-08-04 03:56:23,541 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/acfc709a-44b1-4bca-87d2-f365690f25b6 2025-08-04 03:56:23,580 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/238ce869-cc33-41b5-954e-e354838043dc 2025-08-04 03:56:26,814 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1008ms') 2025-08-04 03:56:26,814 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-33302525 failed! 2025-08-04 03:56:26,832 - DEBUG - Gateway for vn default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 is 214.167.83.193 and allocation pool is NOT set 2025-08-04 03:56:30,910 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1018ms') 2025-08-04 03:56:30,910 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-33302525 failed! 2025-08-04 03:56:30,934 - DEBUG - Gateway for vn default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 is 214.167.83.193 and allocation pool is NOT set 2025-08-04 03:56:35,007 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1015ms') 2025-08-04 03:56:35,007 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-33302525 failed! 2025-08-04 03:56:35,028 - DEBUG - Gateway for vn default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 is 214.167.83.193 and allocation pool is NOT set 2025-08-04 03:56:39,100 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1011ms') 2025-08-04 03:56:39,100 - DEBUG - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-33302525 failed! 2025-08-04 03:56:39,120 - DEBUG - Gateway for vn default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 is 214.167.83.193 and allocation pool is NOT set 2025-08-04 03:56:41,192 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=4.35 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=3.11 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 3.107/3.730/4.354/0.623 ms') 2025-08-04 03:56:41,192 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn1_vm1-33302525 passed 2025-08-04 03:56:41,317 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:56:41,317 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-33302525, IP 214.167.83.195, Port 22 2025-08-04 03:56:41,377 - DEBUG - Error on ssh to cirros@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-04 03:56:41,496 - DEBUG - VM ctest-vn1_vm1-33302525 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-04 03:56:46,497 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:56:46,497 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-33302525, IP 214.167.83.195, Port 22 2025-08-04 03:56:46,561 - DEBUG - Error on ssh to cirros@169.254.0.3:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.3/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.3/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.3/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-04 03:56:46,653 - DEBUG - VM ctest-vn1_vm1-33302525 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-04 03:56:51,654 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:56:51,654 - DEBUG - Waiting to SSH to VM ctest-vn1_vm1-33302525, IP 214.167.83.195, Port 22 2025-08-04 03:56:51,812 - DEBUG - VM ctest-vn1_vm1-33302525 is ready for SSH connections 2025-08-04 03:56:51,812 - INFO - Waiting for VM ctest-vn1_vm2-95359845 to be up.. 2025-08-04 03:56:51,906 - DEBUG - VM is in ACTIVE state now 2025-08-04 03:56:51,906 - INFO - VM name : ctest-vn1_vm2-95359845 2025-08-04 03:56:52,007 - DEBUG - VM ctest-vn1_vm2-95359845 ID is 64a11a6d-cb45-43b8-88a9-91c0eb005a70 2025-08-04 03:56:52,007 - DEBUG - VM ctest-vn1_vm2-95359845 launched on Node an-jenkins-deploy-platform-ansible-os-3750-2 2025-08-04 03:56:52,099 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/64a11a6d-cb45-43b8-88a9-91c0eb005a70 2025-08-04 03:56:52,110 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/59c82ec6-9888-40e9-9468-2ec4cbb18cef 2025-08-04 03:56:53,321 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=1 ttl=63 time=2.98 ms\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=0.978 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1002ms\r\nrtt min/avg/max/mdev = 0.978/1.979/2.981/1.001 ms') 2025-08-04 03:56:53,322 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-vn1_vm2-95359845 passed 2025-08-04 03:56:53,439 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:56:53,439 - DEBUG - Waiting to SSH to VM ctest-vn1_vm2-95359845, IP 214.167.83.196, Port 22 2025-08-04 03:56:53,596 - DEBUG - VM ctest-vn1_vm2-95359845 is ready for SSH connections 2025-08-04 03:56:53,953 - DEBUG - Response for create_network : {'network': {'id': '2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d', 'name': 'ctest-vn2-60124453', 'tenant_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'project_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'ctest-vn2-60124453'], 'port_security_enabled': True, 'description': ''}} 2025-08-04 03:56:54,242 - DEBUG - Response for create_subnet : {'subnet': {'id': '9b4e8e41-f3bf-4647-9138-eee7df38405e', 'name': '', 'tenant_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'network_id': '2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d', 'ip_version': 4, 'cidr': '141.70.171.0/26', 'allocation_pools': [{'start': '141.70.171.2', 'end': '141.70.171.62'}], 'gateway_ip': '141.70.171.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '141.70.171.2', 'tags': [], 'project_id': 'a48ab5304ea04d359e730dcdaaab3e63'}} 2025-08-04 03:56:54,267 - INFO - Created VN ctest-vn2-60124453 2025-08-04 03:56:54,286 - DEBUG - VN ctest-vn2-60124453 UUID is 2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:56:54,701 - DEBUG - Response for create_network : {'network': {'id': '803b3dbe-c1bd-4fc2-9c92-5759bc307503', 'name': 'ctest-fip_vn-14850913', 'tenant_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'project_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'ctest-fip_vn-14850913'], 'port_security_enabled': True, 'description': ''}} 2025-08-04 03:56:55,024 - DEBUG - Response for create_subnet : {'subnet': {'id': 'd3b92695-4602-44e4-a6c4-38f29115b8ca', 'name': '', 'tenant_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'network_id': '803b3dbe-c1bd-4fc2-9c92-5759bc307503', 'ip_version': 4, 'cidr': '174.77.43.192/26', 'allocation_pools': [{'start': '174.77.43.194', 'end': '174.77.43.254'}], 'gateway_ip': '174.77.43.193', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '174.77.43.194', 'tags': [], 'project_id': 'a48ab5304ea04d359e730dcdaaab3e63'}} 2025-08-04 03:56:55,057 - INFO - Created VN ctest-fip_vn-14850913 2025-08-04 03:56:55,078 - DEBUG - VN ctest-fip_vn-14850913 UUID is 803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:56:55,165 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3750-2', 'an-jenkins-deploy-platform-ansible-os-3750-1']} 2025-08-04 03:56:56,495 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3750-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3750-1) 2025-08-04 03:56:58,146 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3750-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3750-1) 2025-08-04 03:56:59,798 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3750-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3750-2) 2025-08-04 03:56:59,799 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:56:59,807 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:56:59,825 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:56:59,945 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:56:59,962 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:56:59,976 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:56:59,986 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:56:59,996 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 03:57:00,005 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:57:00,005 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:00,025 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:57:00,053 - INFO - Verified VN network id 13 for VN f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:00,054 - INFO - Verifications in API Server for VN ctest-vn1-08779962 passed 2025-08-04 03:57:00,054 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:00,085 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:57:00,101 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:57:00,114 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 03:57:00,135 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:56:17', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:42.319552'} 2025-08-04 03:57:00,156 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:57:00,169 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:56:17', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:42.352297'} 2025-08-04 03:57:00,199 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:57:00,211 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:56:17', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:42.394328'} 2025-08-04 03:57:00,223 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:57:00,223 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-08779962 passed 2025-08-04 03:57:00,223 - DEBUG - ====Verifying policy data for ctest-vn1-08779962 in API_Server ====== 2025-08-04 03:57:00,223 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:00,247 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:00,282 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:00,449 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:00,477 - DEBUG - =>VN ctest-vn1-08779962 has no policy to be verified 2025-08-04 03:57:00,477 - DEBUG - Verifying the vn in opserver 2025-08-04 03:57:00,477 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.27 2025-08-04 03:57:00,477 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:57:00,597 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:57:00,597 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.38 2025-08-04 03:57:00,597 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:57:00,676 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:57:00,676 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.61 2025-08-04 03:57:00,676 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:57:00,773 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:57:00,773 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 is found in opserver 2025-08-04 03:57:00,805 - DEBUG - VRF ids for VN ctest-vn1-08779962: {'10.0.0.57': '2'} 2025-08-04 03:57:00,806 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:00,821 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:00,868 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:00,990 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:01,023 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:01,048 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:01,060 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:01,076 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/5c49b08b-a708-46d0-a73c-8dea330b70de 2025-08-04 03:57:01,089 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:57:01,089 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:01,116 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:01,154 - INFO - Verified VN network id 14 for VN 2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:01,154 - INFO - Verifications in API Server for VN ctest-vn2-60124453 passed 2025-08-04 03:57:01,154 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:01,178 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:01,194 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:01,215 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/5c49b08b-a708-46d0-a73c-8dea330b70de 2025-08-04 03:57:01,240 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:07.059930'} 2025-08-04 03:57:01,253 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:57:01,270 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:07.088524'} 2025-08-04 03:57:01,278 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:57:01,288 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:07.104672'} 2025-08-04 03:57:01,297 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:57:01,297 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-60124453 passed 2025-08-04 03:57:01,298 - DEBUG - ====Verifying policy data for ctest-vn2-60124453 in API_Server ====== 2025-08-04 03:57:01,298 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:01,307 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:01,327 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:01,466 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:01,488 - DEBUG - =>VN ctest-vn2-60124453 has no policy to be verified 2025-08-04 03:57:01,488 - DEBUG - Verifying the vn in opserver 2025-08-04 03:57:01,488 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.27 2025-08-04 03:57:01,488 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:57:01,571 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:57:01,571 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.38 2025-08-04 03:57:01,571 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:57:01,588 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:57:01,588 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.61 2025-08-04 03:57:01,588 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:57:01,606 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:57:01,606 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 is found in opserver 2025-08-04 03:57:01,627 - DEBUG - VRF ids for VN ctest-vn2-60124453: {'10.0.0.41': '2'} 2025-08-04 03:57:01,628 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:01,650 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:01,670 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:01,774 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:57:01,795 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:57:01,813 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:57:01,823 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:57:01,839 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/3d83ade0-6dae-4396-ad20-78a064cc6683 2025-08-04 03:57:01,847 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 03:57:01,847 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:57:01,859 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:57:01,883 - INFO - Verified VN network id 15 for VN 803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:57:01,883 - INFO - Verifications in API Server for VN ctest-fip_vn-14850913 passed 2025-08-04 03:57:01,883 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:57:01,897 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:57:01,907 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:57:01,918 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/3d83ade0-6dae-4396-ad20-78a064cc6683 2025-08-04 03:57:01,935 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9240046949984194498', 'uuid-lslong': '11282176059437511939'}, 'enable': 'true', 'created': '2025-08-04T03:56:54', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:07.028736'} 2025-08-04 03:57:01,944 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 03:57:01,961 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9240046949984194498', 'uuid-lslong': '11282176059437511939'}, 'enable': 'true', 'created': '2025-08-04T03:56:54', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:07.062674'} 2025-08-04 03:57:01,975 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 03:57:01,987 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9240046949984194498', 'uuid-lslong': '11282176059437511939'}, 'enable': 'true', 'created': '2025-08-04T03:56:54', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:07.082038'} 2025-08-04 03:57:01,997 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 03:57:01,997 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-fip_vn-14850913 passed 2025-08-04 03:57:01,997 - DEBUG - ====Verifying policy data for ctest-fip_vn-14850913 in API_Server ====== 2025-08-04 03:57:01,997 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:02,008 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:02,032 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:02,136 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:57:02,149 - DEBUG - =>VN ctest-fip_vn-14850913 has no policy to be verified 2025-08-04 03:57:02,149 - DEBUG - Verifying the vn in opserver 2025-08-04 03:57:02,149 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 virtual network link through opserver 10.0.0.27 2025-08-04 03:57:02,149 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:57:02,164 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913?flat'} 2025-08-04 03:57:02,165 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 virtual network link through opserver 10.0.0.38 2025-08-04 03:57:02,165 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:57:02,243 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913?flat'} 2025-08-04 03:57:02,244 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 virtual network link through opserver 10.0.0.61 2025-08-04 03:57:02,244 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:57:02,324 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913?flat'} 2025-08-04 03:57:02,324 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 is found in opserver 2025-08-04 03:57:02,349 - DEBUG - VRF ids for VN ctest-fip_vn-14850913: {'10.0.0.41': '3'} 2025-08-04 03:57:02,349 - INFO - Waiting for VM ctest-fvn_vm1-68714219 to be up.. 2025-08-04 03:57:02,454 - DEBUG - VM is in ACTIVE state now 2025-08-04 03:57:02,455 - INFO - VM name : ctest-fvn_vm1-68714219 2025-08-04 03:57:02,564 - DEBUG - VM ctest-fvn_vm1-68714219 ID is a34e14fe-1542-4253-a0c9-8a5a03c475cf 2025-08-04 03:57:02,565 - DEBUG - VM ctest-fvn_vm1-68714219 launched on Node an-jenkins-deploy-platform-ansible-os-3750-1 2025-08-04 03:57:02,665 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/a34e14fe-1542-4253-a0c9-8a5a03c475cf 2025-08-04 03:57:02,678 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/e9fc5fef-ef4f-40f2-b7de-f52f517506c9 2025-08-04 03:57:05,933 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1033ms') 2025-08-04 03:57:05,933 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-fvn_vm1-68714219 failed! 2025-08-04 03:57:05,953 - DEBUG - Gateway for vn default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 is 174.77.43.193 and allocation pool is NOT set 2025-08-04 03:57:10,025 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1008ms') 2025-08-04 03:57:10,026 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-fvn_vm1-68714219 failed! 2025-08-04 03:57:10,042 - DEBUG - Gateway for vn default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 is 174.77.43.193 and allocation pool is NOT set 2025-08-04 03:57:14,120 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-08-04 03:57:14,120 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-fvn_vm1-68714219 failed! 2025-08-04 03:57:14,139 - DEBUG - Gateway for vn default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 is 174.77.43.193 and allocation pool is NOT set 2025-08-04 03:57:18,218 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1017ms') 2025-08-04 03:57:18,218 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-fvn_vm1-68714219 failed! 2025-08-04 03:57:18,235 - DEBUG - Gateway for vn default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 is 174.77.43.193 and allocation pool is NOT set 2025-08-04 03:57:22,310 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 0 received, 100% packet loss, time 1012ms') 2025-08-04 03:57:22,310 - DEBUG - Ping to Metadata IP 169.254.0.4 of VM ctest-fvn_vm1-68714219 failed! 2025-08-04 03:57:22,329 - DEBUG - Gateway for vn default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 is 174.77.43.193 and allocation pool is NOT set 2025-08-04 03:57:26,413 - DEBUG - (True, 'PING 169.254.0.4 (169.254.0.4) 56(84) bytes of data.\r\n64 bytes from 169.254.0.4: icmp_seq=2 ttl=63 time=7.48 ms\r\n\r\n--- 169.254.0.4 ping statistics ---\r\n2 packets transmitted, 1 received, 50% packet loss, time 1018ms\r\nrtt min/avg/max/mdev = 7.480/7.480/7.480/0.000 ms') 2025-08-04 03:57:26,413 - INFO - Ping to Metadata IP 169.254.0.4 of VM ctest-fvn_vm1-68714219 passed 2025-08-04 03:57:26,535 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:57:26,535 - DEBUG - Waiting to SSH to VM ctest-fvn_vm1-68714219, IP 174.77.43.195, Port 22 2025-08-04 03:57:26,603 - DEBUG - Error on ssh to cirros@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-04 03:57:26,719 - DEBUG - VM ctest-fvn_vm1-68714219 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-04 03:57:31,720 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:57:31,720 - DEBUG - Waiting to SSH to VM ctest-fvn_vm1-68714219, IP 174.77.43.195, Port 22 2025-08-04 03:57:31,788 - DEBUG - Error on ssh to cirros@169.254.0.4:22, result: /bin/bash: connect: Connection refused /bin/bash: line 1: /dev/tcp/169.254.0.4/22: Connection refused {'failed': True, 'command': '(echo > /dev/tcp/169.254.0.4/22)', 'real_command': '/bin/bash -l -c "(echo > /dev/tcp/169.254.0.4/22)"', 'return_code': 1, 'succeeded': False, 'stderr': ''} 2025-08-04 03:57:31,879 - DEBUG - VM ctest-fvn_vm1-68714219 is NOT ready for SSH connections, VM status: ACTIVE 2025-08-04 03:57:36,880 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:57:36,880 - DEBUG - Waiting to SSH to VM ctest-fvn_vm1-68714219, IP 174.77.43.195, Port 22 2025-08-04 03:57:37,048 - DEBUG - VM ctest-fvn_vm1-68714219 is ready for SSH connections 2025-08-04 03:57:37,048 - INFO - Waiting for VM ctest-vn2_vm2-40656987 to be up.. 2025-08-04 03:57:37,158 - DEBUG - VM is in ACTIVE state now 2025-08-04 03:57:37,158 - INFO - VM name : ctest-vn2_vm2-40656987 2025-08-04 03:57:37,261 - DEBUG - VM ctest-vn2_vm2-40656987 ID is 9d06d87f-204a-404f-9e83-ff7953361c58 2025-08-04 03:57:37,261 - DEBUG - VM ctest-vn2_vm2-40656987 launched on Node an-jenkins-deploy-platform-ansible-os-3750-1 2025-08-04 03:57:37,359 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/9d06d87f-204a-404f-9e83-ff7953361c58 2025-08-04 03:57:37,373 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/da444fc9-8e0a-43bf-b485-250058f9c832 2025-08-04 03:57:38,598 - DEBUG - (True, 'PING 169.254.0.3 (169.254.0.3) 56(84) bytes of data.\r\n64 bytes from 169.254.0.3: icmp_seq=1 ttl=63 time=2.36 ms\r\n64 bytes from 169.254.0.3: icmp_seq=2 ttl=63 time=0.722 ms\r\n\r\n--- 169.254.0.3 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.722/1.542/2.363/0.820 ms') 2025-08-04 03:57:38,598 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-vn2_vm2-40656987 passed 2025-08-04 03:57:38,716 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:57:38,716 - DEBUG - Waiting to SSH to VM ctest-vn2_vm2-40656987, IP 141.70.171.3, Port 22 2025-08-04 03:57:38,883 - DEBUG - VM ctest-vn2_vm2-40656987 is ready for SSH connections 2025-08-04 03:57:38,883 - INFO - Waiting for VM mltf_vm to be up.. 2025-08-04 03:57:38,978 - DEBUG - VM is in ACTIVE state now 2025-08-04 03:57:38,978 - INFO - VM name : mltf_vm 2025-08-04 03:57:39,081 - DEBUG - VM mltf_vm ID is abb8a5d7-35b7-4a14-9b7e-b69412c1a1f2 2025-08-04 03:57:39,082 - DEBUG - VM mltf_vm launched on Node an-jenkins-deploy-platform-ansible-os-3750-2 2025-08-04 03:57:39,281 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine/abb8a5d7-35b7-4a14-9b7e-b69412c1a1f2 2025-08-04 03:57:39,295 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/7089ea6c-f925-48a7-bfb4-255e1954489d 2025-08-04 03:57:39,308 - DEBUG - Requesting: http://10.0.0.61:8082/virtual-machine-interface/f6811064-7843-4fd7-bca0-c1b7be2c5a45 2025-08-04 03:57:40,552 - DEBUG - (True, 'PING 169.254.0.5 (169.254.0.5) 56(84) bytes of data.\r\n64 bytes from 169.254.0.5: icmp_seq=1 ttl=63 time=2.23 ms\r\n64 bytes from 169.254.0.5: icmp_seq=2 ttl=63 time=0.599 ms\r\n\r\n--- 169.254.0.5 ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r\nrtt min/avg/max/mdev = 0.599/1.412/2.226/0.813 ms') 2025-08-04 03:57:40,552 - INFO - Ping to Metadata IP 169.254.0.5 of VM mltf_vm passed 2025-08-04 03:57:40,671 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:57:40,672 - DEBUG - Waiting to SSH to VM mltf_vm, IP 214.167.83.197, Port 22 2025-08-04 03:57:40,843 - DEBUG - VM mltf_vm is ready for SSH connections 2025-08-04 03:57:40,961 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:57:40,961 - DEBUG - Waiting to SSH to VM mltf_vm, IP 214.167.83.197, Port 22 2025-08-04 03:57:41,137 - DEBUG - VM mltf_vm is ready for SSH connections 2025-08-04 03:57:41,139 - INFO - ================================================================================ 2025-08-04 03:57:41,139 - INFO - STARTING TEST : test_verify_session_record_table_inter_vn 2025-08-04 03:57:41,139 - INFO - TEST DESCRIPTION : Verify session record table ,generated stats between different vns 1.query and verify number of client session records 2.query and verify number of server session records 3.query with local_ip server_port protocol 4.query with server_port local_ip filter by server_port 5.query with client_port remote_ip filter by client_port Total we get three record limit by 2 6.query with sort_fields 2025-08-04 03:57:42,417 - DEBUG - Skipping xmpp flap check 2025-08-04 03:57:42,417 - INFO - Initial checks done. Running the testcase now 2025-08-04 03:57:42,417 - INFO - 2025-08-04 03:57:42,644 - INFO - Setting flow export rate: 100 2025-08-04 03:57:42,701 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = udp, src_addresses = [subnet = None, virtual_network = any, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = any, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-08-04 03:57:43,061 - DEBUG - Response for mapping policy(s) with vn {'network': {'id': 'f4f2b026-bd63-4f2f-b7d1-2cc78ff80148', 'name': 'ctest-vn1-08779962', 'tenant_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'project_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': ['62dad8db-c361-4be0-ba79-889d78e2cc4f'], 'fq_name': ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'ctest-vn1-08779962'], 'policys': [['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1']], 'subnet_ipam': [{'subnet_cidr': '214.167.83.192/26', 'ipam_fq_name': ['default-domain', 'default-project', 'default-network-ipam']}], 'port_security_enabled': True, 'description': ''}} 2025-08-04 03:57:43,285 - DEBUG - Response for mapping policy(s) with vn {'network': {'id': '2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d', 'name': 'ctest-vn2-60124453', 'tenant_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'project_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': ['9b4e8e41-f3bf-4647-9138-eee7df38405e'], 'fq_name': ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'ctest-vn2-60124453'], 'policys': [['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1']], 'subnet_ipam': [{'subnet_cidr': '141.70.171.0/26', 'ipam_fq_name': ['default-domain', 'default-project', 'default-network-ipam']}], 'port_security_enabled': True, 'description': ''}} 2025-08-04 03:57:43,385 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:43,395 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:43,413 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:43,525 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:43,560 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:43,578 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:57:43,593 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:57:43,604 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 03:57:43,613 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:57:43,613 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:43,628 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:57:43,692 - INFO - Verified VN network id 13 for VN f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:43,692 - INFO - Verifications in API Server for VN ctest-vn1-08779962 passed 2025-08-04 03:57:43,692 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:43,713 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:57:43,739 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:57:43,751 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 03:57:43,790 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:57:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.759805'} 2025-08-04 03:57:43,799 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:57:43,807 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:57:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.773920'} 2025-08-04 03:57:43,816 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:57:43,826 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:57:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.792243'} 2025-08-04 03:57:43,835 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:57:43,835 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-08779962 passed 2025-08-04 03:57:43,836 - DEBUG - ====Verifying policy data for ctest-vn1-08779962 in API_Server ====== 2025-08-04 03:57:43,836 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:43,846 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:43,866 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:43,983 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:43,999 - DEBUG - ==>Verifying data for policy with id: f023407c-63d3-4472-bf75-654fd897b6aa, fqn: ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1'] 2025-08-04 03:57:43,999 - INFO - VN ctest-vn1-08779962 Policy verification: verify_vn_policy_in_api_server, status: True 2025-08-04 03:57:43,999 - DEBUG - Verifying the vn in opserver 2025-08-04 03:57:43,999 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.27 2025-08-04 03:57:43,999 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:57:44,017 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:57:44,017 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.38 2025-08-04 03:57:44,017 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:57:44,034 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:57:44,034 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.61 2025-08-04 03:57:44,034 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:57:44,051 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:57:44,051 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 is found in opserver 2025-08-04 03:57:44,064 - DEBUG - VRF ids for VN ctest-vn1-08779962: {'10.0.0.57': '2'} 2025-08-04 03:57:44,064 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:44,072 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:44,088 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:44,190 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:44,210 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:44,225 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:44,239 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:44,250 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/5c49b08b-a708-46d0-a73c-8dea330b70de 2025-08-04 03:57:44,259 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:57:44,260 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:44,275 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:44,298 - INFO - Verified VN network id 14 for VN 2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:44,298 - INFO - Verifications in API Server for VN ctest-vn2-60124453 passed 2025-08-04 03:57:44,298 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:44,315 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:44,328 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:44,341 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/5c49b08b-a708-46d0-a73c-8dea330b70de 2025-08-04 03:57:44,361 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:57:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.111580'} 2025-08-04 03:57:44,370 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:57:44,380 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:57:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.130167'} 2025-08-04 03:57:44,388 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:57:44,397 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:57:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.159901'} 2025-08-04 03:57:44,407 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:57:44,407 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-60124453 passed 2025-08-04 03:57:44,407 - DEBUG - ====Verifying policy data for ctest-vn2-60124453 in API_Server ====== 2025-08-04 03:57:44,408 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:44,416 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:44,432 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:44,537 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:44,554 - DEBUG - ==>Verifying data for policy with id: f023407c-63d3-4472-bf75-654fd897b6aa, fqn: ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1'] 2025-08-04 03:57:44,554 - INFO - VN ctest-vn2-60124453 Policy verification: verify_vn_policy_in_api_server, status: True 2025-08-04 03:57:44,554 - DEBUG - Verifying the vn in opserver 2025-08-04 03:57:44,554 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.27 2025-08-04 03:57:44,554 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:57:44,571 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:57:44,571 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.38 2025-08-04 03:57:44,571 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:57:44,589 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:57:44,589 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.61 2025-08-04 03:57:44,589 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:57:44,607 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:57:44,607 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 is found in opserver 2025-08-04 03:57:44,627 - DEBUG - VRF ids for VN ctest-vn2-60124453: {'10.0.0.41': '2', '10.0.0.57': '3'} 2025-08-04 03:57:44,628 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:44,640 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:44,660 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:44,764 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:44,779 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:44,792 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:57:44,815 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:57:44,832 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 03:57:44,843 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:57:44,843 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:44,860 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:57:44,892 - INFO - Verified VN network id 13 for VN f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:44,892 - INFO - Verifications in API Server for VN ctest-vn1-08779962 passed 2025-08-04 03:57:44,892 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:44,912 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:57:44,924 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:57:44,939 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 03:57:44,960 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:57:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.928655'} 2025-08-04 03:57:44,970 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:57:44,976 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:57:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.943860'} 2025-08-04 03:57:44,983 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:57:44,991 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:57:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.957377'} 2025-08-04 03:57:45,001 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:57:45,001 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-08779962 passed 2025-08-04 03:57:45,001 - DEBUG - ====Verifying policy data for ctest-vn1-08779962 in API_Server ====== 2025-08-04 03:57:45,001 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:45,010 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:45,025 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:45,141 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:57:45,156 - DEBUG - ==>Verifying data for policy with id: f023407c-63d3-4472-bf75-654fd897b6aa, fqn: ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1'] 2025-08-04 03:57:45,156 - INFO - VN ctest-vn1-08779962 Policy verification: verify_vn_policy_in_api_server, status: True 2025-08-04 03:57:45,156 - DEBUG - Verifying the vn in opserver 2025-08-04 03:57:45,156 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.27 2025-08-04 03:57:45,156 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:57:45,174 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:57:45,174 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.38 2025-08-04 03:57:45,174 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:57:45,194 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:57:45,194 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.61 2025-08-04 03:57:45,194 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:57:45,210 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:57:45,210 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 is found in opserver 2025-08-04 03:57:45,226 - DEBUG - VRF ids for VN ctest-vn1-08779962: {'10.0.0.57': '2'} 2025-08-04 03:57:45,226 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:45,235 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:45,252 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:45,357 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:45,379 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:45,395 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:45,407 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:45,418 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/5c49b08b-a708-46d0-a73c-8dea330b70de 2025-08-04 03:57:45,429 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:57:45,429 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:45,444 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:45,467 - INFO - Verified VN network id 14 for VN 2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:45,467 - INFO - Verifications in API Server for VN ctest-vn2-60124453 passed 2025-08-04 03:57:45,467 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:45,487 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:45,506 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:57:45,526 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/5c49b08b-a708-46d0-a73c-8dea330b70de 2025-08-04 03:57:45,546 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:57:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.296516'} 2025-08-04 03:57:45,557 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:57:45,564 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:57:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.315597'} 2025-08-04 03:57:45,572 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:57:45,579 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:57:43', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.343030'} 2025-08-04 03:57:45,588 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:57:45,588 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-60124453 passed 2025-08-04 03:57:45,588 - DEBUG - ====Verifying policy data for ctest-vn2-60124453 in API_Server ====== 2025-08-04 03:57:45,588 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:45,597 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:45,614 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:45,720 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:57:45,734 - DEBUG - ==>Verifying data for policy with id: f023407c-63d3-4472-bf75-654fd897b6aa, fqn: ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1'] 2025-08-04 03:57:45,734 - INFO - VN ctest-vn2-60124453 Policy verification: verify_vn_policy_in_api_server, status: True 2025-08-04 03:57:45,734 - DEBUG - Verifying the vn in opserver 2025-08-04 03:57:45,734 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.27 2025-08-04 03:57:45,734 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:57:45,750 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:57:45,750 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.38 2025-08-04 03:57:45,750 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:57:45,765 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:57:45,765 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.61 2025-08-04 03:57:45,765 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:57:45,783 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:57:45,783 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 is found in opserver 2025-08-04 03:57:45,797 - DEBUG - VRF ids for VN ctest-vn2-60124453: {'10.0.0.41': '2', '10.0.0.57': '3'} 2025-08-04 03:57:45,797 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:45,808 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:45,832 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:45,950 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:57:45,968 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:57:45,982 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:57:45,995 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:57:46,009 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/3d83ade0-6dae-4396-ad20-78a064cc6683 2025-08-04 03:57:46,024 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 03:57:46,024 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:57:46,042 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:57:46,072 - INFO - Verified VN network id 15 for VN 803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:57:46,072 - INFO - Verifications in API Server for VN ctest-fip_vn-14850913 passed 2025-08-04 03:57:46,073 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:57:46,087 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:57:46,101 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:57:46,118 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/3d83ade0-6dae-4396-ad20-78a064cc6683 2025-08-04 03:57:46,140 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9240046949984194498', 'uuid-lslong': '11282176059437511939'}, 'enable': 'true', 'created': '2025-08-04T03:56:54', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:51.231530'} 2025-08-04 03:57:46,151 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 03:57:46,157 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9240046949984194498', 'uuid-lslong': '11282176059437511939'}, 'enable': 'true', 'created': '2025-08-04T03:56:54', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:51.259054'} 2025-08-04 03:57:46,164 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 03:57:46,172 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9240046949984194498', 'uuid-lslong': '11282176059437511939'}, 'enable': 'true', 'created': '2025-08-04T03:56:54', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:51.270423'} 2025-08-04 03:57:46,181 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 03:57:46,181 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-fip_vn-14850913 passed 2025-08-04 03:57:46,181 - DEBUG - ====Verifying policy data for ctest-fip_vn-14850913 in API_Server ====== 2025-08-04 03:57:46,181 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:57:46,188 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:57:46,204 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:57:46,297 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:57:46,313 - DEBUG - =>VN ctest-fip_vn-14850913 has no policy to be verified 2025-08-04 03:57:46,313 - DEBUG - Verifying the vn in opserver 2025-08-04 03:57:46,313 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 virtual network link through opserver 10.0.0.27 2025-08-04 03:57:46,313 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:57:46,333 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913?flat'} 2025-08-04 03:57:46,333 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 virtual network link through opserver 10.0.0.38 2025-08-04 03:57:46,333 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:57:46,350 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913?flat'} 2025-08-04 03:57:46,350 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 virtual network link through opserver 10.0.0.61 2025-08-04 03:57:46,351 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:57:46,366 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913?flat'} 2025-08-04 03:57:46,366 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 is found in opserver 2025-08-04 03:57:46,379 - DEBUG - VRF ids for VN ctest-fip_vn-14850913: {'10.0.0.41': '3'} 2025-08-04 03:57:46,379 - INFO - start time= 1754279866379543 2025-08-04 03:57:46,379 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:57:50,166 - DEBUG - file copy failed retrying 2025-08-04 03:57:53,574 - DEBUG - file copy failed retrying 2025-08-04 03:57:56,977 - DEBUG - file copy failed retrying 2025-08-04 03:58:00,397 - DEBUG - file copy failed retrying 2025-08-04 03:58:00,398 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:58:00,398 - DEBUG - Copied file /contrail-test/tcutils/traffic_utils/../udpechoserver.py to 141.70.171.3 2025-08-04 03:58:00,398 - DEBUG - Running remote_cmd, Cmd : python /tmp/udpechoserver.py --start_port 9000 --end_port 9000 --pid_file /tmp/server-ctest-random-36556747.pid --stats_file /tmp/server-ctest-random-36556747.stats 0<&- &> /tmp/server-ctest-random-36556747.log, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.41, gateway password: c0ntrail123 2025-08-04 03:58:00,398 - DEBUG - nohup python /tmp/udpechoserver.py --start_port 9000 --end_port 9000 --pid_file /tmp/server-ctest-random-36556747.pid --stats_file /tmp/server-ctest-random-36556747.stats 0<&- &> /tmp/server-ctest-random-36556747.log & 2025-08-04 03:58:01,013 - DEBUG - None 2025-08-04 03:58:04,666 - DEBUG - file copy failed retrying 2025-08-04 03:58:08,106 - DEBUG - file copy failed retrying 2025-08-04 03:58:11,470 - DEBUG - file copy failed retrying 2025-08-04 03:58:14,862 - DEBUG - file copy failed retrying 2025-08-04 03:58:14,862 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:58:14,863 - DEBUG - Copied file /contrail-test/tcutils/traffic_utils/../udpechoclient.py to 214.167.83.195 2025-08-04 03:58:14,863 - DEBUG - Running remote_cmd, Cmd : python /tmp/udpechoclient.py --servers 141.70.171.3 --sport 8000 --dports 9000 --retry --count 100 --pid_file /tmp/client-ctest-random-36556747.pid --stats_file /tmp/client-ctest-random-36556747.stats 0<&- &> /tmp/client-ctest-random-36556747.log, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-08-04 03:58:14,863 - DEBUG - nohup python /tmp/udpechoclient.py --servers 141.70.171.3 --sport 8000 --dports 9000 --retry --count 100 --pid_file /tmp/client-ctest-random-36556747.pid --stats_file /tmp/client-ctest-random-36556747.stats 0<&- &> /tmp/client-ctest-random-36556747.log & 2025-08-04 03:58:15,424 - DEBUG - None 2025-08-04 03:58:15,424 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:58:15,424 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:58:15,424 - DEBUG - File /contrail-test/tcutils/traffic_utils/../udpechoserver.py already copied to 141.70.171.3 2025-08-04 03:58:15,424 - DEBUG - Running remote_cmd, Cmd : python /tmp/udpechoserver.py --start_port 9001 --end_port 9001 --pid_file /tmp/server-ctest-random-49283030.pid --stats_file /tmp/server-ctest-random-49283030.stats 0<&- &> /tmp/server-ctest-random-49283030.log, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.41, gateway password: c0ntrail123 2025-08-04 03:58:15,425 - DEBUG - nohup python /tmp/udpechoserver.py --start_port 9001 --end_port 9001 --pid_file /tmp/server-ctest-random-49283030.pid --stats_file /tmp/server-ctest-random-49283030.stats 0<&- &> /tmp/server-ctest-random-49283030.log & 2025-08-04 03:58:16,043 - DEBUG - None 2025-08-04 03:58:16,043 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:58:16,043 - DEBUG - File /contrail-test/tcutils/traffic_utils/../udpechoclient.py already copied to 214.167.83.195 2025-08-04 03:58:16,043 - DEBUG - Running remote_cmd, Cmd : python /tmp/udpechoclient.py --servers 141.70.171.3 --sport 8000 --dports 9001 --retry --count 100 --pid_file /tmp/client-ctest-random-49283030.pid --stats_file /tmp/client-ctest-random-49283030.stats 0<&- &> /tmp/client-ctest-random-49283030.log, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-08-04 03:58:16,043 - DEBUG - nohup python /tmp/udpechoclient.py --servers 141.70.171.3 --sport 8000 --dports 9001 --retry --count 100 --pid_file /tmp/client-ctest-random-49283030.pid --stats_file /tmp/client-ctest-random-49283030.stats 0<&- &> /tmp/client-ctest-random-49283030.log & 2025-08-04 03:58:16,250 - DEBUG - None 2025-08-04 03:58:16,250 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:58:16,250 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:58:16,250 - DEBUG - File /contrail-test/tcutils/traffic_utils/../udpechoserver.py already copied to 141.70.171.3 2025-08-04 03:58:16,250 - DEBUG - Running remote_cmd, Cmd : python /tmp/udpechoserver.py --start_port 9003 --end_port 9003 --pid_file /tmp/server-ctest-random-41069126.pid --stats_file /tmp/server-ctest-random-41069126.stats 0<&- &> /tmp/server-ctest-random-41069126.log, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.41, gateway password: c0ntrail123 2025-08-04 03:58:16,250 - DEBUG - nohup python /tmp/udpechoserver.py --start_port 9003 --end_port 9003 --pid_file /tmp/server-ctest-random-41069126.pid --stats_file /tmp/server-ctest-random-41069126.stats 0<&- &> /tmp/server-ctest-random-41069126.log & 2025-08-04 03:58:16,526 - DEBUG - None 2025-08-04 03:58:16,527 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 03:58:16,527 - DEBUG - File /contrail-test/tcutils/traffic_utils/../udpechoclient.py already copied to 214.167.83.195 2025-08-04 03:58:16,527 - DEBUG - Running remote_cmd, Cmd : python /tmp/udpechoclient.py --servers 141.70.171.3 --sport 8000 --dports 9003 --retry --count 100 --pid_file /tmp/client-ctest-random-41069126.pid --stats_file /tmp/client-ctest-random-41069126.stats 0<&- &> /tmp/client-ctest-random-41069126.log, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-08-04 03:58:16,527 - DEBUG - nohup python /tmp/udpechoclient.py --servers 141.70.171.3 --sport 8000 --dports 9003 --retry --count 100 --pid_file /tmp/client-ctest-random-41069126.pid --stats_file /tmp/client-ctest-random-41069126.stats 0<&- &> /tmp/client-ctest-random-41069126.log & 2025-08-04 03:58:16,764 - DEBUG - None 2025-08-04 03:58:26,765 - DEBUG - Running remote_cmd, Cmd : kill $(cat /tmp/client-ctest-random-36556747.pid); sync; cat /tmp/client-ctest-random-36556747.stats, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-08-04 03:58:26,765 - DEBUG - kill $(cat /tmp/client-ctest-random-36556747.pid); sync; cat /tmp/client-ctest-random-36556747.stats 2025-08-04 03:58:27,117 - DEBUG - cat: can't open '/tmp/client-ctest-random-36556747.pid': No such file or directory sh: you need to specify whom to kill cat: can't open '/tmp/client-ctest-random-36556747.stats': No such file or directory 2025-08-04 03:58:27,118 - DEBUG - Running remote_cmd, Cmd : kill $(cat /tmp/server-ctest-random-36556747.pid); sync; cat /tmp/server-ctest-random-36556747.stats, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.41, gateway password: c0ntrail123 2025-08-04 03:58:27,118 - DEBUG - kill $(cat /tmp/server-ctest-random-36556747.pid); sync; cat /tmp/server-ctest-random-36556747.stats 2025-08-04 03:58:27,490 - DEBUG - cat: can't open '/tmp/server-ctest-random-36556747.pid': No such file or directory sh: you need to specify whom to kill cat: can't open '/tmp/server-ctest-random-36556747.stats': No such file or directory 2025-08-04 03:58:27,490 - INFO - Client - Sent: 0, Received: 0 2025-08-04 03:58:27,490 - INFO - Server - Sent: 0, Received: 0 2025-08-04 03:58:27,934 - INFO - Deleted policy policy1 2025-08-04 03:58:28,108 - INFO - Setting flow export rate: None 2025-08-04 03:58:30,946 - ERROR - AssertionError Python 3.6.8: /usr/bin/python3 Mon Aug 4 03:58:27 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/serial_scripts/analytics/test_analytics_resource.py in test_verify_session_record_table_inter_vn(self=) 1373 start_time = self.analytics_obj.getstarttime(vm_node_ip) 1374 self.logger.info("start time= %s" % (start_time)) 1375 self.setup_and_create_streams(self.res.vn1_vm1_fixture, self.res.vn2_vm2_fixture) 1376 time.sleep(100) 1377 src_vn = self.res.vn1_fixture.vn_fq_name self = self.setup_and_create_streams = > self.res = self.res.vn1_vm1_fixture = self.res.vn2_vm2_fixture = /contrail-test/serial_scripts/analytics/base.py in setup_and_create_streams(self=, src_vm=, dst_vm=, sport=8000, dport=9003, count=100) 168 time.sleep(10) 169 for traffic_obj in traffic_objs: 170 self.stop_traffic(traffic_obj) 171 172 #end setup_create_streams self = self.stop_traffic = > traffic_obj = /contrail-test/common/base.py in stop_traffic(self=, traffic_obj=, expectation=True, unidirection=False, partial=False) 796 assert False, "No drops observed in packets %s"%msg 797 elif expectation: 798 assert sent and recv, "Packets not %s"%msg 799 if recv*100/float(sent) < 90: 800 assert False, "Packets not %s"%msg sent = 0 recv = 0 msg = 'transferred between 214.167.83.195 and 141.70.171.3, proto udp sport 8000 and dport 9000' AssertionError: Packets not transferred between 214.167.83.195 and 141.70.171.3, proto udp sport 8000 and dport 9000 __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('Packets not transferred between 214.167.83.195 and 141.70.171.3, proto udp sport 8000 and dport 9000',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/serial_scripts/analytics/test_analytics_resource.py", line 1375, in test_verify_session_record_table_inter_vn self.setup_and_create_streams(self.res.vn1_vm1_fixture, self.res.vn2_vm2_fixture) File "/contrail-test/serial_scripts/analytics/base.py", line 170, in setup_and_create_streams self.stop_traffic(traffic_obj) File "/contrail-test/common/base.py", line 798, in stop_traffic assert sent and recv, "Packets not %s"%msg AssertionError: Packets not transferred between 214.167.83.195 and 141.70.171.3, proto udp sport 8000 and dport 9000 2025-08-04 03:58:30,946 - DEBUG - Skipping xmpp flap check 2025-08-04 03:58:30,946 - INFO - 2025-08-04 03:58:30,947 - INFO - END TEST : test_verify_session_record_table_inter_vn : FAILED[0:00:49] 2025-08-04 03:58:30,947 - INFO - -------------------------------------------------------------------------------- 2025-08-04 03:58:30,956 - INFO - ================================================================================ 2025-08-04 03:58:30,956 - INFO - STARTING TEST : test_verify_session_sampling_teardown 2025-08-04 03:58:30,956 - INFO - TEST DESCRIPTION : 1.query client session samples 2.query server session samples 3.query client session to get number of sessions exported 4.query session record table for teardown bytes/pkts 5.query sample count after teardown on server side 2025-08-04 03:58:32,244 - DEBUG - Skipping xmpp flap check 2025-08-04 03:58:32,245 - INFO - Initial checks done. Running the testcase now 2025-08-04 03:58:32,245 - INFO - 2025-08-04 03:58:32,464 - INFO - Setting flow export rate: 100 2025-08-04 03:58:32,512 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = icmp, src_addresses = [subnet = None, virtual_network = any, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = any, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-08-04 03:58:32,891 - DEBUG - Response for mapping policy(s) with vn {'network': {'id': 'f4f2b026-bd63-4f2f-b7d1-2cc78ff80148', 'name': 'ctest-vn1-08779962', 'tenant_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'project_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': ['62dad8db-c361-4be0-ba79-889d78e2cc4f'], 'fq_name': ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'ctest-vn1-08779962'], 'policys': [['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1']], 'subnet_ipam': [{'subnet_cidr': '214.167.83.192/26', 'ipam_fq_name': ['default-domain', 'default-project', 'default-network-ipam']}], 'port_security_enabled': True, 'description': ''}} 2025-08-04 03:58:33,189 - DEBUG - Response for mapping policy(s) with vn {'network': {'id': '2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d', 'name': 'ctest-vn2-60124453', 'tenant_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'project_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': ['9b4e8e41-f3bf-4647-9138-eee7df38405e'], 'fq_name': ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'ctest-vn2-60124453'], 'policys': [['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1']], 'subnet_ipam': [{'subnet_cidr': '141.70.171.0/26', 'ipam_fq_name': ['default-domain', 'default-project', 'default-network-ipam']}], 'port_security_enabled': True, 'description': ''}} 2025-08-04 03:58:33,348 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:58:33,356 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:58:33,370 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:58:33,471 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:58:33,499 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:58:33,519 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:58:33,534 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:58:33,548 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 03:58:33,559 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:58:33,559 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:58:33,572 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:58:33,597 - INFO - Verified VN network id 13 for VN f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:58:33,598 - INFO - Verifications in API Server for VN ctest-vn1-08779962 passed 2025-08-04 03:58:33,598 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:58:33,616 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:58:33,630 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:58:33,645 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 03:58:33,665 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:58:32', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.825900'} 2025-08-04 03:58:33,675 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:58:33,683 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:58:32', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.844380'} 2025-08-04 03:58:33,693 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:58:33,702 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:58:32', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.864094'} 2025-08-04 03:58:33,713 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:58:33,713 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-08779962 passed 2025-08-04 03:58:33,713 - DEBUG - ====Verifying policy data for ctest-vn1-08779962 in API_Server ====== 2025-08-04 03:58:33,713 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:58:33,722 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:58:33,745 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:58:33,852 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:58:33,871 - DEBUG - ==>Verifying data for policy with id: 72d67ac5-2721-4bbe-9543-d92285436821, fqn: ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1'] 2025-08-04 03:58:33,871 - INFO - VN ctest-vn1-08779962 Policy verification: verify_vn_policy_in_api_server, status: True 2025-08-04 03:58:33,871 - DEBUG - Verifying the vn in opserver 2025-08-04 03:58:33,871 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.27 2025-08-04 03:58:33,871 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:58:33,892 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:58:33,892 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.38 2025-08-04 03:58:33,892 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:58:33,908 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:58:33,908 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.61 2025-08-04 03:58:33,908 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:58:33,927 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:58:33,927 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 is found in opserver 2025-08-04 03:58:33,941 - DEBUG - VRF ids for VN ctest-vn1-08779962: {'10.0.0.57': '2'} 2025-08-04 03:58:33,942 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:58:33,949 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:58:33,982 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:58:34,091 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:58:34,112 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:58:34,127 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:58:34,144 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:58:34,154 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/5c49b08b-a708-46d0-a73c-8dea330b70de 2025-08-04 03:58:34,167 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:58:34,167 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:58:34,182 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:58:34,209 - INFO - Verified VN network id 14 for VN 2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:58:34,210 - INFO - Verifications in API Server for VN ctest-vn2-60124453 passed 2025-08-04 03:58:34,210 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:58:34,228 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:58:34,240 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:58:34,252 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/5c49b08b-a708-46d0-a73c-8dea330b70de 2025-08-04 03:58:34,274 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:58:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.153003'} 2025-08-04 03:58:34,286 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:58:34,294 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:58:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.175769'} 2025-08-04 03:58:34,305 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:58:34,313 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:58:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.192660'} 2025-08-04 03:58:34,325 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:58:34,326 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-60124453 passed 2025-08-04 03:58:34,326 - DEBUG - ====Verifying policy data for ctest-vn2-60124453 in API_Server ====== 2025-08-04 03:58:34,326 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:58:34,334 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:58:34,351 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:58:34,462 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:58:34,477 - DEBUG - ==>Verifying data for policy with id: 72d67ac5-2721-4bbe-9543-d92285436821, fqn: ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1'] 2025-08-04 03:58:34,477 - INFO - VN ctest-vn2-60124453 Policy verification: verify_vn_policy_in_api_server, status: True 2025-08-04 03:58:34,478 - DEBUG - Verifying the vn in opserver 2025-08-04 03:58:34,478 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.27 2025-08-04 03:58:34,478 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:58:34,494 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:58:34,494 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.38 2025-08-04 03:58:34,494 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:58:34,511 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:58:34,511 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.61 2025-08-04 03:58:34,511 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:58:34,535 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:58:34,535 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 is found in opserver 2025-08-04 03:58:34,551 - DEBUG - VRF ids for VN ctest-vn2-60124453: {'10.0.0.41': '2', '10.0.0.57': '3'} 2025-08-04 03:58:34,551 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:58:34,560 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:58:34,579 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:58:34,682 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:58:34,709 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:58:34,727 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:58:34,738 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:58:34,748 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 03:58:34,761 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:58:34,761 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:58:34,776 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:58:34,801 - INFO - Verified VN network id 13 for VN f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:58:34,801 - INFO - Verifications in API Server for VN ctest-vn1-08779962 passed 2025-08-04 03:58:34,802 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:58:34,816 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:58:34,828 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 03:58:34,843 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 03:58:34,873 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:58:32', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.028725'} 2025-08-04 03:58:34,886 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:58:34,895 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:58:32', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.055930'} 2025-08-04 03:58:34,903 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:58:34,918 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T03:58:32', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.079984'} 2025-08-04 03:58:34,930 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 03:58:34,930 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-08779962 passed 2025-08-04 03:58:34,930 - DEBUG - ====Verifying policy data for ctest-vn1-08779962 in API_Server ====== 2025-08-04 03:58:34,930 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:58:34,945 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:58:34,963 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:58:35,073 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 03:58:35,087 - DEBUG - ==>Verifying data for policy with id: 72d67ac5-2721-4bbe-9543-d92285436821, fqn: ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1'] 2025-08-04 03:58:35,088 - INFO - VN ctest-vn1-08779962 Policy verification: verify_vn_policy_in_api_server, status: True 2025-08-04 03:58:35,088 - DEBUG - Verifying the vn in opserver 2025-08-04 03:58:35,088 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.27 2025-08-04 03:58:35,088 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:58:35,106 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:58:35,106 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.38 2025-08-04 03:58:35,106 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:58:35,121 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:58:35,121 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.61 2025-08-04 03:58:35,121 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:58:35,137 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 03:58:35,137 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 is found in opserver 2025-08-04 03:58:35,155 - DEBUG - VRF ids for VN ctest-vn1-08779962: {'10.0.0.57': '2'} 2025-08-04 03:58:35,155 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:58:35,163 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:58:35,183 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:58:35,288 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:58:35,304 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:58:35,318 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:58:35,329 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:58:35,340 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/5c49b08b-a708-46d0-a73c-8dea330b70de 2025-08-04 03:58:35,350 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:58:35,350 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:58:35,366 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:58:35,386 - INFO - Verified VN network id 14 for VN 2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:58:35,386 - INFO - Verifications in API Server for VN ctest-vn2-60124453 passed 2025-08-04 03:58:35,386 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:58:35,404 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:58:35,415 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 03:58:35,426 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/5c49b08b-a708-46d0-a73c-8dea330b70de 2025-08-04 03:58:35,444 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:58:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.325116'} 2025-08-04 03:58:35,455 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:58:35,462 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:58:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.344779'} 2025-08-04 03:58:35,470 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:58:35,478 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T03:58:33', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.357245'} 2025-08-04 03:58:35,489 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 03:58:35,489 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-60124453 passed 2025-08-04 03:58:35,490 - DEBUG - ====Verifying policy data for ctest-vn2-60124453 in API_Server ====== 2025-08-04 03:58:35,490 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:58:35,498 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:58:35,515 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:58:35,635 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 03:58:35,651 - DEBUG - ==>Verifying data for policy with id: 72d67ac5-2721-4bbe-9543-d92285436821, fqn: ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1'] 2025-08-04 03:58:35,651 - INFO - VN ctest-vn2-60124453 Policy verification: verify_vn_policy_in_api_server, status: True 2025-08-04 03:58:35,651 - DEBUG - Verifying the vn in opserver 2025-08-04 03:58:35,652 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.27 2025-08-04 03:58:35,652 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:58:35,668 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:58:35,668 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.38 2025-08-04 03:58:35,668 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:58:35,684 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:58:35,685 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.61 2025-08-04 03:58:35,685 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:58:35,703 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 03:58:35,703 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 is found in opserver 2025-08-04 03:58:35,721 - DEBUG - VRF ids for VN ctest-vn2-60124453: {'10.0.0.41': '2', '10.0.0.57': '3'} 2025-08-04 03:58:35,722 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:58:35,732 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:58:35,753 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:58:35,880 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:58:35,898 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:58:35,924 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:58:35,934 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:58:35,945 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/3d83ade0-6dae-4396-ad20-78a064cc6683 2025-08-04 03:58:35,963 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 03:58:35,963 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:58:35,979 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:58:36,001 - INFO - Verified VN network id 15 for VN 803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:58:36,002 - INFO - Verifications in API Server for VN ctest-fip_vn-14850913 passed 2025-08-04 03:58:36,002 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:58:36,019 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:58:36,029 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 03:58:36,040 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/3d83ade0-6dae-4396-ad20-78a064cc6683 2025-08-04 03:58:36,066 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9240046949984194498', 'uuid-lslong': '11282176059437511939'}, 'enable': 'true', 'created': '2025-08-04T03:56:54', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:41.156577'} 2025-08-04 03:58:36,077 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 03:58:36,087 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9240046949984194498', 'uuid-lslong': '11282176059437511939'}, 'enable': 'true', 'created': '2025-08-04T03:56:54', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:41.188381'} 2025-08-04 03:58:36,098 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 03:58:36,108 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9240046949984194498', 'uuid-lslong': '11282176059437511939'}, 'enable': 'true', 'created': '2025-08-04T03:56:54', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:01:41.205624'} 2025-08-04 03:58:36,124 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 03:58:36,124 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-fip_vn-14850913 passed 2025-08-04 03:58:36,124 - DEBUG - ====Verifying policy data for ctest-fip_vn-14850913 in API_Server ====== 2025-08-04 03:58:36,124 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 03:58:36,134 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 03:58:36,152 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 03:58:36,273 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 03:58:36,288 - DEBUG - =>VN ctest-fip_vn-14850913 has no policy to be verified 2025-08-04 03:58:36,288 - DEBUG - Verifying the vn in opserver 2025-08-04 03:58:36,288 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 virtual network link through opserver 10.0.0.27 2025-08-04 03:58:36,288 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 03:58:36,306 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913?flat'} 2025-08-04 03:58:36,306 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 virtual network link through opserver 10.0.0.38 2025-08-04 03:58:36,306 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 03:58:36,324 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913?flat'} 2025-08-04 03:58:36,324 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 virtual network link through opserver 10.0.0.61 2025-08-04 03:58:36,324 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 03:58:36,351 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913?flat'} 2025-08-04 03:58:36,351 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 is found in opserver 2025-08-04 03:58:36,369 - DEBUG - VRF ids for VN ctest-fip_vn-14850913: {'10.0.0.41': '3'} 2025-08-04 03:58:36,369 - INFO - start time= 1754279916369510 2025-08-04 03:58:36,369 - DEBUG - Running remote_cmd, Cmd : ping -s 56 -c 3 -W 1 141.70.171.3, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-08-04 03:58:36,370 - DEBUG - ping -s 56 -c 3 -W 1 141.70.171.3 2025-08-04 03:58:38,615 - DEBUG - PING 141.70.171.3 (141.70.171.3): 56 data bytes 64 bytes from 141.70.171.3: seq=0 ttl=63 time=4.495 ms 64 bytes from 141.70.171.3: seq=1 ttl=63 time=1.235 ms 64 bytes from 141.70.171.3: seq=2 ttl=63 time=1.365 ms --- 141.70.171.3 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 1.235/2.365/4.495 ms 2025-08-04 03:58:38,615 - INFO - Ping to IP 141.70.171.3 from VM ctest-vn1_vm1-33302525 passed 2025-08-04 03:58:48,616 - INFO - Verify session samples and teardown pkts 2025-08-04 03:58:48,616 - INFO - Verifying SessionSeriesTable through opserver 10.0.0.27 2025-08-04 03:58:48,620 - DEBUG - Posting: http://10.0.0.27:8081/analytics/query, payload {'table': 'SessionSeriesTable', 'start_time': 1754279916369510, 'end_time': 1754279928620834, 'select_fields': ['T'], 'where': [[{'name': 'vn', 'value': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'op': 1, 'value2': None}, {'name': 'remote_vn', 'value': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'op': 1, 'value2': None}, {'name': 'protocol', 'value': '1', 'op': 1, 'value2': None}]], 'session_type': 'client'} 2025-08-04 03:58:49,642 - DEBUG - [{'T': 1754279916909067}, {'T': 1754279917909171}, {'T': 1754279918909218}] 2025-08-04 03:58:49,643 - DEBUG - Posting: http://10.0.0.27:8081/analytics/query, payload {'table': 'SessionSeriesTable', 'start_time': 1754279916369510, 'end_time': 1754279929643008, 'select_fields': ['T'], 'where': [[{'name': 'vn', 'value': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'op': 1, 'value2': None}, {'name': 'remote_vn', 'value': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'op': 1, 'value2': None}, {'name': 'protocol', 'value': '1', 'op': 1, 'value2': None}]], 'session_type': 'server'} 2025-08-04 03:58:50,664 - DEBUG - [{'T': 1754279917533069}, {'T': 1754279918533115}, {'T': 1754279919533170}] 2025-08-04 03:58:50,664 - DEBUG - Posting: http://10.0.0.27:8081/analytics/query, payload {'table': 'SessionSeriesTable', 'start_time': 1754279916369510, 'end_time': 1754279930664704, 'select_fields': ['vn', 'remote_vn', 'sample_count'], 'where': [[{'name': 'vn', 'value': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'op': 1, 'value2': None}, {'name': 'remote_vn', 'value': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'op': 1, 'value2': None}, {'name': 'protocol', 'value': '1', 'op': 1, 'value2': None}]], 'session_type': 'client'} 2025-08-04 03:58:51,688 - DEBUG - [{'remote_vn': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'vn': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'sample_count': 3}] 2025-08-04 03:58:51,688 - INFO - wait for the flows to get expire 2025-08-04 04:02:11,689 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3750-2:Compute:contrail-vrouter-agent:0?flat 2025-08-04 04:02:11,799 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3750-2?flat 2025-08-04 04:02:11,896 - DEBUG - Posting: http://10.0.0.27:8081/analytics/query, payload {'table': 'SessionRecordTable', 'start_time': 1754279916369510, 'end_time': 1754280131896595, 'select_fields': ['vn', 'remote_vn', 'forward_teardown_pkts', 'reverse_teardown_pkts'], 'where': [[{'name': 'vn', 'value': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'op': 1, 'value2': None}, {'name': 'remote_vn', 'value': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'op': 1, 'value2': None}, {'name': 'protocol', 'value': '1', 'op': 1, 'value2': None}]], 'session_type': 'client'} 2025-08-04 04:02:12,971 - DEBUG - [{'forward_flow_uuid': '60fd0725-67eb-4fb2-8c01-69dc5896c994', 'remote_vn': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'reverse_flow_uuid': '759a27fe-3856-41a4-8d56-9bd95daee444', 'vn': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'forward_teardown_pkts': 3, 'reverse_teardown_pkts': 3}] 2025-08-04 04:02:12,971 - DEBUG - Posting: http://10.0.0.27:8081/analytics/query, payload {'table': 'SessionSeriesTable', 'start_time': 1754279916369510, 'end_time': 1754280132971532, 'select_fields': ['vn', 'remote_vn', 'sample_count'], 'where': [[{'name': 'vn', 'value': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'op': 1, 'value2': None}, {'name': 'remote_vn', 'value': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'op': 1, 'value2': None}, {'name': 'protocol', 'value': '1', 'op': 1, 'value2': None}]], 'session_type': 'client'} 2025-08-04 04:02:14,054 - DEBUG - Posting: http://10.0.0.27:8081/analytics/query, payload {'table': 'SessionSeriesTable', 'start_time': 1754279916369510, 'end_time': 1754280134054059, 'select_fields': ['vn', 'remote_vn', 'sample_count'], 'where': [[{'name': 'vn', 'value': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'op': 1, 'value2': None}, {'name': 'remote_vn', 'value': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'op': 1, 'value2': None}, {'name': 'protocol', 'value': '1', 'op': 1, 'value2': None}]], 'session_type': 'server'} 2025-08-04 04:02:15,086 - DEBUG - [{'remote_vn': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'vn': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'sample_count': 4}] 2025-08-04 04:02:15,702 - INFO - Deleted policy policy1 2025-08-04 04:02:15,860 - INFO - Setting flow export rate: None 2025-08-04 04:02:17,144 - DEBUG - Skipping xmpp flap check 2025-08-04 04:02:17,144 - INFO - END TEST : test_verify_session_sampling_teardown : PASSED[0:03:47] 2025-08-04 04:02:17,144 - INFO - -------------------------------------------------------------------------------- 2025-08-04 04:02:17,148 - INFO - ================================================================================ 2025-08-04 04:02:17,148 - INFO - STARTING TEST : test_verify_session_series_table_inter_vn 2025-08-04 04:02:17,148 - INFO - TEST DESCRIPTION : Verify session series table ,generated stats between different vns 1.query client session samples 2.query for server ports 3.sort results by server_port column 4.verify granularity with T=10 5.verify sampled bytes 6.verify logged bytes 7.query and verify filter by action 2025-08-04 04:02:18,442 - DEBUG - Skipping xmpp flap check 2025-08-04 04:02:18,442 - INFO - Initial checks done. Running the testcase now 2025-08-04 04:02:18,442 - INFO - 2025-08-04 04:02:18,660 - INFO - Setting flow export rate: 100 2025-08-04 04:02:18,708 - DEBUG - Policy np_rules : [rule_sequence = None, rule_uuid = None, direction = <>, protocol = udp, src_addresses = [subnet = None, virtual_network = any, security_group = None, network_policy = None, subnet_list = []], src_ports = [start_port = -1, end_port = -1], application = [], dst_addresses = [subnet = None, virtual_network = any, security_group = None, network_policy = None, subnet_list = []], dst_ports = [start_port = -1, end_port = -1], action_list = simple_action = pass, gateway_name = None, apply_service = [], service_properties = None, mirror_to = None, assign_routing_instance = None, log = False, alert = False, qos_action = None, host_based_service = False, ethertype = None, created = None, last_modified = None, description = None] 2025-08-04 04:02:19,115 - DEBUG - Response for mapping policy(s) with vn {'network': {'id': 'f4f2b026-bd63-4f2f-b7d1-2cc78ff80148', 'name': 'ctest-vn1-08779962', 'tenant_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'project_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': ['62dad8db-c361-4be0-ba79-889d78e2cc4f'], 'fq_name': ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'ctest-vn1-08779962'], 'policys': [['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1']], 'subnet_ipam': [{'subnet_cidr': '214.167.83.192/26', 'ipam_fq_name': ['default-domain', 'default-project', 'default-network-ipam']}], 'port_security_enabled': True, 'description': ''}} 2025-08-04 04:02:19,356 - DEBUG - Response for mapping policy(s) with vn {'network': {'id': '2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d', 'name': 'ctest-vn2-60124453', 'tenant_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'project_id': 'a48ab5304ea04d359e730dcdaaab3e63', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': ['9b4e8e41-f3bf-4647-9138-eee7df38405e'], 'fq_name': ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'ctest-vn2-60124453'], 'policys': [['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1']], 'subnet_ipam': [{'subnet_cidr': '141.70.171.0/26', 'ipam_fq_name': ['default-domain', 'default-project', 'default-network-ipam']}], 'port_security_enabled': True, 'description': ''}} 2025-08-04 04:02:19,460 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 04:02:19,522 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 04:02:19,583 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 04:02:19,695 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 04:02:19,717 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 04:02:19,736 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 04:02:19,749 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 04:02:19,765 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 04:02:19,776 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 04:02:19,776 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 04:02:19,791 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 04:02:19,816 - INFO - Verified VN network id 13 for VN f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 04:02:19,816 - INFO - Verifications in API Server for VN ctest-vn1-08779962 passed 2025-08-04 04:02:19,816 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 04:02:19,831 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 04:02:19,845 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 04:02:19,857 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 04:02:19,873 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T04:02:19', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.810447'} 2025-08-04 04:02:19,881 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 04:02:19,889 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T04:02:19', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.826287'} 2025-08-04 04:02:19,899 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 04:02:19,907 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T04:02:19', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.843232'} 2025-08-04 04:02:19,917 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 04:02:19,917 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-08779962 passed 2025-08-04 04:02:19,917 - DEBUG - ====Verifying policy data for ctest-vn1-08779962 in API_Server ====== 2025-08-04 04:02:19,917 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 04:02:19,925 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 04:02:19,941 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 04:02:20,052 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 04:02:20,065 - DEBUG - ==>Verifying data for policy with id: 34d84b98-dac0-43ea-b8f6-fac071ae686d, fqn: ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1'] 2025-08-04 04:02:20,065 - INFO - VN ctest-vn1-08779962 Policy verification: verify_vn_policy_in_api_server, status: True 2025-08-04 04:02:20,065 - DEBUG - Verifying the vn in opserver 2025-08-04 04:02:20,065 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.27 2025-08-04 04:02:20,065 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 04:02:20,083 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 04:02:20,083 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.38 2025-08-04 04:02:20,083 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 04:02:20,161 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 04:02:20,161 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.61 2025-08-04 04:02:20,161 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 04:02:20,226 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 04:02:20,226 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 is found in opserver 2025-08-04 04:02:20,240 - DEBUG - VRF ids for VN ctest-vn1-08779962: {'10.0.0.57': '2'} 2025-08-04 04:02:20,240 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 04:02:20,249 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 04:02:20,264 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 04:02:20,369 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 04:02:20,389 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 04:02:20,403 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 04:02:20,417 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 04:02:20,429 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/5c49b08b-a708-46d0-a73c-8dea330b70de 2025-08-04 04:02:20,441 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 04:02:20,441 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 04:02:20,456 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 04:02:20,480 - INFO - Verified VN network id 14 for VN 2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 04:02:20,480 - INFO - Verifications in API Server for VN ctest-vn2-60124453 passed 2025-08-04 04:02:20,480 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 04:02:20,499 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 04:02:20,512 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 04:02:20,524 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/5c49b08b-a708-46d0-a73c-8dea330b70de 2025-08-04 04:02:20,552 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T04:02:19', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.238169'} 2025-08-04 04:02:20,565 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 04:02:20,575 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T04:02:19', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.263801'} 2025-08-04 04:02:20,593 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 04:02:20,604 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T04:02:19', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.292281'} 2025-08-04 04:02:20,617 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 04:02:20,617 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-60124453 passed 2025-08-04 04:02:20,617 - DEBUG - ====Verifying policy data for ctest-vn2-60124453 in API_Server ====== 2025-08-04 04:02:20,617 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 04:02:20,626 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 04:02:20,647 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 04:02:20,756 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 04:02:20,772 - DEBUG - ==>Verifying data for policy with id: 34d84b98-dac0-43ea-b8f6-fac071ae686d, fqn: ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1'] 2025-08-04 04:02:20,772 - INFO - VN ctest-vn2-60124453 Policy verification: verify_vn_policy_in_api_server, status: True 2025-08-04 04:02:20,772 - DEBUG - Verifying the vn in opserver 2025-08-04 04:02:20,772 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.27 2025-08-04 04:02:20,772 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 04:02:20,788 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 04:02:20,789 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.38 2025-08-04 04:02:20,789 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 04:02:20,850 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 04:02:20,850 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.61 2025-08-04 04:02:20,850 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 04:02:20,916 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 04:02:20,916 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 is found in opserver 2025-08-04 04:02:20,934 - DEBUG - VRF ids for VN ctest-vn2-60124453: {'10.0.0.41': '2', '10.0.0.57': '3'} 2025-08-04 04:02:20,934 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 04:02:20,943 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 04:02:20,960 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 04:02:21,065 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 04:02:21,078 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 04:02:21,093 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 04:02:21,104 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 04:02:21,116 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 04:02:21,125 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 04:02:21,125 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 04:02:21,145 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 04:02:21,170 - INFO - Verified VN network id 13 for VN f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 04:02:21,170 - INFO - Verifications in API Server for VN ctest-vn1-08779962 passed 2025-08-04 04:02:21,170 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 04:02:21,184 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 04:02:21,196 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 04:02:21,209 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/ef14356b-eec4-430c-a633-8e3efe436700 2025-08-04 04:02:21,230 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T04:02:19', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.166748'} 2025-08-04 04:02:21,239 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 04:02:21,247 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T04:02:19', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.182926'} 2025-08-04 04:02:21,255 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 04:02:21,263 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17650363570051043119', 'uuid-lslong': '13245417214698848584'}, 'enable': 'true', 'created': '2025-08-04T03:56:17', 'last-modified': '2025-08-04T04:02:19', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.199012'} 2025-08-04 04:02:21,271 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-08-04 04:02:21,271 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn1-08779962 passed 2025-08-04 04:02:21,272 - DEBUG - ====Verifying policy data for ctest-vn1-08779962 in API_Server ====== 2025-08-04 04:02:21,272 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 04:02:21,281 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 04:02:21,296 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 04:02:21,400 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 2025-08-04 04:02:21,413 - DEBUG - ==>Verifying data for policy with id: 34d84b98-dac0-43ea-b8f6-fac071ae686d, fqn: ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1'] 2025-08-04 04:02:21,414 - INFO - VN ctest-vn1-08779962 Policy verification: verify_vn_policy_in_api_server, status: True 2025-08-04 04:02:21,414 - DEBUG - Verifying the vn in opserver 2025-08-04 04:02:21,414 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.27 2025-08-04 04:02:21,414 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 04:02:21,431 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 04:02:21,431 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.38 2025-08-04 04:02:21,431 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 04:02:21,490 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 04:02:21,490 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 virtual network link through opserver 10.0.0.61 2025-08-04 04:02:21,490 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 04:02:21,507 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962?flat'} 2025-08-04 04:02:21,507 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 is found in opserver 2025-08-04 04:02:21,524 - DEBUG - VRF ids for VN ctest-vn1-08779962: {'10.0.0.57': '2'} 2025-08-04 04:02:21,525 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 04:02:21,532 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 04:02:21,548 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 04:02:21,650 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 04:02:21,663 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 04:02:21,677 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 04:02:21,688 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 04:02:21,698 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/5c49b08b-a708-46d0-a73c-8dea330b70de 2025-08-04 04:02:21,706 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 04:02:21,706 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 04:02:21,719 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 04:02:21,740 - INFO - Verified VN network id 14 for VN 2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 04:02:21,741 - INFO - Verifications in API Server for VN ctest-vn2-60124453 passed 2025-08-04 04:02:21,741 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 04:02:21,755 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 04:02:21,767 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 04:02:21,779 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/5c49b08b-a708-46d0-a73c-8dea330b70de 2025-08-04 04:02:21,801 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T04:02:19', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.489648'} 2025-08-04 04:02:21,811 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 04:02:21,818 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T04:02:19', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.508856'} 2025-08-04 04:02:21,825 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 04:02:21,833 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '3129156126332044862', 'uuid-lslong': '13085947531724407853'}, 'enable': 'true', 'created': '2025-08-04T03:56:53', 'last-modified': '2025-08-04T04:02:19', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:02.523621'} 2025-08-04 04:02:21,842 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-08-04 04:02:21,842 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn2-60124453 passed 2025-08-04 04:02:21,842 - DEBUG - ====Verifying policy data for ctest-vn2-60124453 in API_Server ====== 2025-08-04 04:02:21,842 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 04:02:21,851 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 04:02:21,867 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 04:02:21,976 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/2b6cfeea-fcc2-463e-b59a-9ff2a1d5442d 2025-08-04 04:02:21,990 - DEBUG - ==>Verifying data for policy with id: 34d84b98-dac0-43ea-b8f6-fac071ae686d, fqn: ['default-domain', 'ctest-AnalyticsTestSanityWithResource-33294378', 'policy1'] 2025-08-04 04:02:21,990 - INFO - VN ctest-vn2-60124453 Policy verification: verify_vn_policy_in_api_server, status: True 2025-08-04 04:02:21,990 - DEBUG - Verifying the vn in opserver 2025-08-04 04:02:21,990 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.27 2025-08-04 04:02:21,991 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 04:02:22,007 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 04:02:22,007 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.38 2025-08-04 04:02:22,007 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 04:02:22,025 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 04:02:22,025 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 virtual network link through opserver 10.0.0.61 2025-08-04 04:02:22,025 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 04:02:22,044 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453?flat'} 2025-08-04 04:02:22,044 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 is found in opserver 2025-08-04 04:02:22,057 - DEBUG - VRF ids for VN ctest-vn2-60124453: {'10.0.0.41': '2', '10.0.0.57': '3'} 2025-08-04 04:02:22,057 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 04:02:22,065 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 04:02:22,081 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 04:02:22,188 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 04:02:22,202 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 04:02:22,216 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 04:02:22,227 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 04:02:22,237 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/3d83ade0-6dae-4396-ad20-78a064cc6683 2025-08-04 04:02:22,246 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 04:02:22,246 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 04:02:22,257 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 04:02:22,281 - INFO - Verified VN network id 15 for VN 803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 04:02:22,281 - INFO - Verifications in API Server for VN ctest-fip_vn-14850913 passed 2025-08-04 04:02:22,281 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 04:02:22,300 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 04:02:22,314 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 04:02:22,324 - DEBUG - Requesting: http://10.0.0.27:8082/route-target/3d83ade0-6dae-4396-ad20-78a064cc6683 2025-08-04 04:02:22,342 - DEBUG - Control-node 10.0.0.27 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9240046949984194498', 'uuid-lslong': '11282176059437511939'}, 'enable': 'true', 'created': '2025-08-04T03:56:54', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:05:27.435096'} 2025-08-04 04:02:22,352 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 04:02:22,358 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9240046949984194498', 'uuid-lslong': '11282176059437511939'}, 'enable': 'true', 'created': '2025-08-04T03:56:54', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:05:27.460211'} 2025-08-04 04:02:22,366 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 04:02:22,374 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'interests': '0', 'advertised': '0', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-AnalyticsTestSanityWithResource-33294378', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '9240046949984194498', 'uuid-lslong': '11282176059437511939'}, 'enable': 'true', 'created': '2025-08-04T03:56:54', 'last-modified': '2025-08-04T03:56:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:05:27.471204'} 2025-08-04 04:02:22,385 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-08-04 04:02:22,385 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-fip_vn-14850913 passed 2025-08-04 04:02:22,385 - DEBUG - ====Verifying policy data for ctest-fip_vn-14850913 in API_Server ====== 2025-08-04 04:02:22,385 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 04:02:22,393 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 04:02:22,411 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 04:02:22,528 - DEBUG - Requesting: http://10.0.0.27:8082/virtual-network/803b3dbe-c1bd-4fc2-9c92-5759bc307503 2025-08-04 04:02:22,548 - DEBUG - =>VN ctest-fip_vn-14850913 has no policy to be verified 2025-08-04 04:02:22,548 - DEBUG - Verifying the vn in opserver 2025-08-04 04:02:22,548 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 virtual network link through opserver 10.0.0.27 2025-08-04 04:02:22,548 - DEBUG - Requesting: http://10.0.0.27:8081/analytics/uves/virtual-networks 2025-08-04 04:02:22,567 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'href': 'http://10.0.0.27:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913?flat'} 2025-08-04 04:02:22,567 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 virtual network link through opserver 10.0.0.38 2025-08-04 04:02:22,567 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-08-04 04:02:22,582 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913?flat'} 2025-08-04 04:02:22,582 - DEBUG - Verifying the default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 virtual network link through opserver 10.0.0.61 2025-08-04 04:02:22,583 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-08-04 04:02:22,601 - DEBUG - vn link and name as {'name': 'default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913?flat'} 2025-08-04 04:02:22,601 - INFO - Validated that VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 is found in opserver 2025-08-04 04:02:22,620 - DEBUG - VRF ids for VN ctest-fip_vn-14850913: {'10.0.0.41': '3'} 2025-08-04 04:02:22,620 - INFO - start time= 1754280142620279 2025-08-04 04:02:22,620 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 04:02:22,620 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 04:02:22,620 - DEBUG - File /contrail-test/tcutils/traffic_utils/../udpechoserver.py already copied to 141.70.171.3 2025-08-04 04:02:22,620 - DEBUG - Running remote_cmd, Cmd : python /tmp/udpechoserver.py --start_port 9000 --end_port 9000 --pid_file /tmp/server-ctest-random-60301236.pid --stats_file /tmp/server-ctest-random-60301236.stats 0<&- &> /tmp/server-ctest-random-60301236.log, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.41, gateway password: c0ntrail123 2025-08-04 04:02:22,620 - DEBUG - nohup python /tmp/udpechoserver.py --start_port 9000 --end_port 9000 --pid_file /tmp/server-ctest-random-60301236.pid --stats_file /tmp/server-ctest-random-60301236.stats 0<&- &> /tmp/server-ctest-random-60301236.log & 2025-08-04 04:02:22,875 - DEBUG - None 2025-08-04 04:02:22,876 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 04:02:22,876 - DEBUG - File /contrail-test/tcutils/traffic_utils/../udpechoclient.py already copied to 214.167.83.195 2025-08-04 04:02:22,876 - DEBUG - Running remote_cmd, Cmd : python /tmp/udpechoclient.py --servers 141.70.171.3 --sport 8000 --dports 9000 --retry --count 100 --pid_file /tmp/client-ctest-random-60301236.pid --stats_file /tmp/client-ctest-random-60301236.stats 0<&- &> /tmp/client-ctest-random-60301236.log, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-08-04 04:02:22,876 - DEBUG - nohup python /tmp/udpechoclient.py --servers 141.70.171.3 --sport 8000 --dports 9000 --retry --count 100 --pid_file /tmp/client-ctest-random-60301236.pid --stats_file /tmp/client-ctest-random-60301236.stats 0<&- &> /tmp/client-ctest-random-60301236.log & 2025-08-04 04:02:23,095 - DEBUG - None 2025-08-04 04:02:23,095 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 04:02:23,096 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 04:02:23,096 - DEBUG - File /contrail-test/tcutils/traffic_utils/../udpechoserver.py already copied to 141.70.171.3 2025-08-04 04:02:23,096 - DEBUG - Running remote_cmd, Cmd : python /tmp/udpechoserver.py --start_port 9001 --end_port 9001 --pid_file /tmp/server-ctest-random-97693672.pid --stats_file /tmp/server-ctest-random-97693672.stats 0<&- &> /tmp/server-ctest-random-97693672.log, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.41, gateway password: c0ntrail123 2025-08-04 04:02:23,096 - DEBUG - nohup python /tmp/udpechoserver.py --start_port 9001 --end_port 9001 --pid_file /tmp/server-ctest-random-97693672.pid --stats_file /tmp/server-ctest-random-97693672.stats 0<&- &> /tmp/server-ctest-random-97693672.log & 2025-08-04 04:02:23,357 - DEBUG - None 2025-08-04 04:02:23,357 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 04:02:23,357 - DEBUG - File /contrail-test/tcutils/traffic_utils/../udpechoclient.py already copied to 214.167.83.195 2025-08-04 04:02:23,357 - DEBUG - Running remote_cmd, Cmd : python /tmp/udpechoclient.py --servers 141.70.171.3 --sport 8000 --dports 9001 --retry --count 100 --pid_file /tmp/client-ctest-random-97693672.pid --stats_file /tmp/client-ctest-random-97693672.stats 0<&- &> /tmp/client-ctest-random-97693672.log, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-08-04 04:02:23,358 - DEBUG - nohup python /tmp/udpechoclient.py --servers 141.70.171.3 --sport 8000 --dports 9001 --retry --count 100 --pid_file /tmp/client-ctest-random-97693672.pid --stats_file /tmp/client-ctest-random-97693672.stats 0<&- &> /tmp/client-ctest-random-97693672.log & 2025-08-04 04:02:23,609 - DEBUG - None 2025-08-04 04:02:23,609 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 04:02:23,610 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 04:02:23,610 - DEBUG - File /contrail-test/tcutils/traffic_utils/../udpechoserver.py already copied to 141.70.171.3 2025-08-04 04:02:23,610 - DEBUG - Running remote_cmd, Cmd : python /tmp/udpechoserver.py --start_port 9003 --end_port 9003 --pid_file /tmp/server-ctest-random-26811757.pid --stats_file /tmp/server-ctest-random-26811757.stats 0<&- &> /tmp/server-ctest-random-26811757.log, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.41, gateway password: c0ntrail123 2025-08-04 04:02:23,610 - DEBUG - nohup python /tmp/udpechoserver.py --start_port 9003 --end_port 9003 --pid_file /tmp/server-ctest-random-26811757.pid --stats_file /tmp/server-ctest-random-26811757.stats 0<&- &> /tmp/server-ctest-random-26811757.log & 2025-08-04 04:02:23,840 - DEBUG - None 2025-08-04 04:02:23,840 - DEBUG - verify_vm_launched is skipped since self.vm_launch_flag is set 2025-08-04 04:02:23,840 - DEBUG - File /contrail-test/tcutils/traffic_utils/../udpechoclient.py already copied to 214.167.83.195 2025-08-04 04:02:23,840 - DEBUG - Running remote_cmd, Cmd : python /tmp/udpechoclient.py --servers 141.70.171.3 --sport 8000 --dports 9003 --retry --count 100 --pid_file /tmp/client-ctest-random-26811757.pid --stats_file /tmp/client-ctest-random-26811757.stats 0<&- &> /tmp/client-ctest-random-26811757.log, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-08-04 04:02:23,840 - DEBUG - nohup python /tmp/udpechoclient.py --servers 141.70.171.3 --sport 8000 --dports 9003 --retry --count 100 --pid_file /tmp/client-ctest-random-26811757.pid --stats_file /tmp/client-ctest-random-26811757.stats 0<&- &> /tmp/client-ctest-random-26811757.log & 2025-08-04 04:02:24,116 - DEBUG - None 2025-08-04 04:02:34,117 - DEBUG - Running remote_cmd, Cmd : kill $(cat /tmp/client-ctest-random-60301236.pid); sync; cat /tmp/client-ctest-random-60301236.stats, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.57, gateway password: c0ntrail123 2025-08-04 04:02:34,117 - DEBUG - kill $(cat /tmp/client-ctest-random-60301236.pid); sync; cat /tmp/client-ctest-random-60301236.stats 2025-08-04 04:02:34,425 - DEBUG - cat: can't open '/tmp/client-ctest-random-60301236.pid': No such file or directory sh: you need to specify whom to kill cat: can't open '/tmp/client-ctest-random-60301236.stats': No such file or directory 2025-08-04 04:02:34,426 - DEBUG - Running remote_cmd, Cmd : kill $(cat /tmp/server-ctest-random-60301236.pid); sync; cat /tmp/server-ctest-random-60301236.stats, host_string: cirros@169.254.0.3, password: gocubsgogateway: ubuntu@10.0.0.41, gateway password: c0ntrail123 2025-08-04 04:02:34,426 - DEBUG - kill $(cat /tmp/server-ctest-random-60301236.pid); sync; cat /tmp/server-ctest-random-60301236.stats 2025-08-04 04:02:34,750 - DEBUG - cat: can't open '/tmp/server-ctest-random-60301236.pid': No such file or directory sh: you need to specify whom to kill cat: can't open '/tmp/server-ctest-random-60301236.stats': No such file or directory 2025-08-04 04:02:34,750 - INFO - Client - Sent: 0, Received: 0 2025-08-04 04:02:34,750 - INFO - Server - Sent: 0, Received: 0 2025-08-04 04:02:35,401 - INFO - Deleted policy policy1 2025-08-04 04:02:35,580 - INFO - Setting flow export rate: None 2025-08-04 04:02:36,864 - ERROR - AssertionError Python 3.6.8: /usr/bin/python3 Mon Aug 4 04:02:34 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/serial_scripts/analytics/test_analytics_resource.py in test_verify_session_series_table_inter_vn(self=) 1336 start_time = self.analytics_obj.getstarttime(vm_node_ip) 1337 self.logger.info("start time= %s" % (start_time)) 1338 self.setup_and_create_streams(self.res.vn1_vm1_fixture, self.res.vn2_vm2_fixture) 1339 src_vn = self.res.vn1_fixture.vn_fq_name 1340 dst_vn = self.res.vn2_fixture.vn_fq_name self = self.setup_and_create_streams = > self.res = self.res.vn1_vm1_fixture = self.res.vn2_vm2_fixture = /contrail-test/serial_scripts/analytics/base.py in setup_and_create_streams(self=, src_vm=, dst_vm=, sport=8000, dport=9003, count=100) 168 time.sleep(10) 169 for traffic_obj in traffic_objs: 170 self.stop_traffic(traffic_obj) 171 172 #end setup_create_streams self = self.stop_traffic = > traffic_obj = /contrail-test/common/base.py in stop_traffic(self=, traffic_obj=, expectation=True, unidirection=False, partial=False) 796 assert False, "No drops observed in packets %s"%msg 797 elif expectation: 798 assert sent and recv, "Packets not %s"%msg 799 if recv*100/float(sent) < 90: 800 assert False, "Packets not %s"%msg sent = 0 recv = 0 msg = 'transferred between 214.167.83.195 and 141.70.171.3, proto udp sport 8000 and dport 9000' AssertionError: Packets not transferred between 214.167.83.195 and 141.70.171.3, proto udp sport 8000 and dport 9000 __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('Packets not transferred between 214.167.83.195 and 141.70.171.3, proto udp sport 8000 and dport 9000',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/serial_scripts/analytics/test_analytics_resource.py", line 1338, in test_verify_session_series_table_inter_vn self.setup_and_create_streams(self.res.vn1_vm1_fixture, self.res.vn2_vm2_fixture) File "/contrail-test/serial_scripts/analytics/base.py", line 170, in setup_and_create_streams self.stop_traffic(traffic_obj) File "/contrail-test/common/base.py", line 798, in stop_traffic assert sent and recv, "Packets not %s"%msg AssertionError: Packets not transferred between 214.167.83.195 and 141.70.171.3, proto udp sport 8000 and dport 9000 2025-08-04 04:02:36,865 - DEBUG - Skipping xmpp flap check 2025-08-04 04:02:36,865 - INFO - 2025-08-04 04:02:36,865 - INFO - END TEST : test_verify_session_series_table_inter_vn : FAILED[0:00:19] 2025-08-04 04:02:36,866 - INFO - -------------------------------------------------------------------------------- 2025-08-04 04:02:36,869 - INFO - Deleting VM mltf_vm 2025-08-04 04:02:36,977 - INFO - Deleting VM ctest-fvn_vm1-68714219 2025-08-04 04:02:37,074 - INFO - Deleting VM ctest-vn2_vm2-40656987 2025-08-04 04:02:37,185 - INFO - Deleting VN ctest-fip_vn-14850913 2025-08-04 04:02:37,255 - DEBUG - VN 803b3dbe-c1bd-4fc2-9c92-5759bc307503 still in use: Unable to complete operation on network 803b3dbe-c1bd-4fc2-9c92-5759bc307503. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-ce1e0fae-f07c-4dad-914b-aa58dea96b26'] 2025-08-04 04:02:37,255 - WARNING - Deleting VN ctest-fip_vn-14850913 failed..Will retry 2025-08-04 04:02:39,486 - DEBUG - Response for deleting network () 2025-08-04 04:02:39,486 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e2f98916-810f-46ff-b3a0-8d9cb6c4d1d1 2025-08-04 04:02:39,509 - DEBUG - Response Code: 404 2025-08-04 04:02:39,509 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 04:02:39,523 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 04:02:39,539 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 04:02:39,633 - INFO - Validated that VN ctest-fip_vn-14850913 is not found in API Server 2025-08-04 04:02:39,648 - DEBUG - VN ctest-fip_vn-14850913 is not present in Agent 10.0.0.41 2025-08-04 04:02:39,660 - DEBUG - VN ctest-fip_vn-14850913 is not present in Agent 10.0.0.57 2025-08-04 04:02:39,660 - INFO - Validated that VN ctest-fip_vn-14850913 is not in any agent 2025-08-04 04:02:39,668 - DEBUG - VRF 3 is not seen in agent 10.0.0.41 2025-08-04 04:02:39,677 - DEBUG - Vrouter 10.0.0.41 does not have vrf 3 for VN ctest-fip_vn-14850913 2025-08-04 04:02:39,677 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-fip_vn-14850913 2025-08-04 04:02:39,706 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-fip_vn-14850913 info 2025-08-04 04:02:39,706 - INFO - Deleting VN ctest-vn2-60124453 2025-08-04 04:02:39,892 - DEBUG - Response for deleting network () 2025-08-04 04:02:39,892 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/30e0c87d-9965-4af0-9284-aaf0f0fe02c3 2025-08-04 04:02:39,912 - DEBUG - Response Code: 404 2025-08-04 04:02:39,912 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 04:02:39,923 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 04:02:39,944 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 04:02:40,047 - INFO - Validated that VN ctest-vn2-60124453 is not found in API Server 2025-08-04 04:02:40,061 - DEBUG - VN ctest-vn2-60124453 is not present in Agent 10.0.0.41 2025-08-04 04:02:40,074 - DEBUG - VN ctest-vn2-60124453 is not present in Agent 10.0.0.57 2025-08-04 04:02:40,074 - INFO - Validated that VN ctest-vn2-60124453 is not in any agent 2025-08-04 04:02:40,082 - DEBUG - VRF 2 is not seen in agent 10.0.0.41 2025-08-04 04:02:40,090 - DEBUG - Vrouter 10.0.0.41 does not have vrf 2 for VN ctest-vn2-60124453 2025-08-04 04:02:40,097 - DEBUG - VRF 3 is not seen in agent 10.0.0.57 2025-08-04 04:02:40,111 - DEBUG - Vrouter 10.0.0.57 does not have vrf 3 for VN ctest-vn2-60124453 2025-08-04 04:02:40,111 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn2-60124453 2025-08-04 04:02:40,145 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn2-60124453 info 2025-08-04 04:02:40,145 - INFO - Deleting VM ctest-vn1_vm2-95359845 2025-08-04 04:02:40,234 - INFO - Deleting VM ctest-vn1_vm1-33302525 2025-08-04 04:02:40,316 - INFO - Deleting VN ctest-vn1-08779962 2025-08-04 04:02:40,369 - DEBUG - VN f4f2b026-bd63-4f2f-b7d1-2cc78ff80148 still in use: Unable to complete operation on network f4f2b026-bd63-4f2f-b7d1-2cc78ff80148. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-6a8444e3-1bea-4b69-a4db-367b71bf90d6'] 2025-08-04 04:02:40,369 - WARNING - Deleting VN ctest-vn1-08779962 failed..Will retry 2025-08-04 04:02:42,597 - DEBUG - Response for deleting network () 2025-08-04 04:02:42,597 - DEBUG - Requesting: http://10.0.0.27:8082/routing-instance/e99be9e0-5910-4043-aa77-dbeaa2b7ac52 2025-08-04 04:02:42,621 - DEBUG - Response Code: 404 2025-08-04 04:02:42,621 - DEBUG - Requesting: http://10.0.0.27:8082/domains 2025-08-04 04:02:42,629 - DEBUG - Requesting: http://10.0.0.27:8082/domain/8d00e278-1e00-4abc-a644-cc23d080d5d1 2025-08-04 04:02:42,647 - DEBUG - Requesting: http://10.0.0.27:8082/project/a48ab530-4ea0-4d35-9e73-0dcdaaab3e63 2025-08-04 04:02:42,752 - INFO - Validated that VN ctest-vn1-08779962 is not found in API Server 2025-08-04 04:02:42,769 - DEBUG - VN ctest-vn1-08779962 is not present in Agent 10.0.0.41 2025-08-04 04:02:42,783 - DEBUG - VN ctest-vn1-08779962 is not present in Agent 10.0.0.57 2025-08-04 04:02:42,784 - INFO - Validated that VN ctest-vn1-08779962 is not in any agent 2025-08-04 04:02:42,792 - DEBUG - VRF 2 is not seen in agent 10.0.0.57 2025-08-04 04:02:42,800 - DEBUG - Vrouter 10.0.0.57 does not have vrf 2 for VN ctest-vn1-08779962 2025-08-04 04:02:42,800 - INFO - Validated that all vrouters do not have the route table for VN default-domain:ctest-AnalyticsTestSanityWithResource-33294378:ctest-vn1-08779962 2025-08-04 04:02:42,837 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn1-08779962 info 2025-08-04 04:02:43,674 - INFO - Deleted project: ctest-AnalyticsTestSanityWithResource-33294378, ID : a48ab530-4ea0-4d35-9e73-0dcdaaab3e63