2025-07-31 04:22:44,077 - INFO - Domain Default found not creating 2025-07-31 04:22:44,448 - INFO - Project ctest-TestSvcHC-45474793 not found, creating it 2025-07-31 04:22:45,000 - INFO - Created Project:ctest-TestSvcHC-45474793, ID : 5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:22:47,292 - INFO - ================================================================================ 2025-07-31 04:22:47,292 - INFO - STARTING TEST : test_svc_hc_basic 2025-07-31 04:22:48,673 - DEBUG - Nothing to compare xmpp stats {'10.0.0.27': {'10.20.0.193': '0', '10.20.0.18': '0'}, '10.0.0.57': {'10.20.0.18': '0', '10.20.0.193': '0'}} with 2025-07-31 04:22:48,673 - INFO - Initial checks done. Running the testcase now 2025-07-31 04:22:48,674 - INFO - 2025-07-31 04:22:49,760 - DEBUG - Response for create_network : {'network': {'id': '4d92304a-da03-4368-b3c9-3e27ce2b1066', 'name': 'ctest-mgmt_vn-64400332', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'project_id': '5d8e29c3f9ed4135910480d12bc16305', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcHC-45474793', 'ctest-mgmt_vn-64400332'], 'port_security_enabled': True, 'description': ''}} 2025-07-31 04:22:50,017 - DEBUG - Response for create_subnet : {'subnet': {'id': '419df8a3-36fb-4e38-baa0-402d03f21090', 'name': '', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'network_id': '4d92304a-da03-4368-b3c9-3e27ce2b1066', 'ip_version': 4, 'cidr': '202.104.129.128/26', 'allocation_pools': [{'start': '202.104.129.130', 'end': '202.104.129.190'}], 'gateway_ip': '202.104.129.129', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '202.104.129.130', 'tags': [], 'project_id': '5d8e29c3f9ed4135910480d12bc16305'}} 2025-07-31 04:22:50,041 - INFO - Created VN ctest-mgmt_vn-64400332 2025-07-31 04:22:50,054 - DEBUG - VN ctest-mgmt_vn-64400332 UUID is 4d92304a-da03-4368-b3c9-3e27ce2b1066 2025-07-31 04:22:50,142 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:22:50,478 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:22:50,553 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:22:50,582 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:22:50,712 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/4d92304a-da03-4368-b3c9-3e27ce2b1066 2025-07-31 04:22:50,729 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/4d92304a-da03-4368-b3c9-3e27ce2b1066 2025-07-31 04:22:50,742 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/4cea2f01-fb49-4c7f-9e2e-05610ea766a1 2025-07-31 04:22:50,757 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/4cea2f01-fb49-4c7f-9e2e-05610ea766a1 2025-07-31 04:22:50,769 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/6fcf8a1e-49f9-4b4c-b7e8-e2a9ef40b02a 2025-07-31 04:22:50,784 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 04:22:50,784 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/4d92304a-da03-4368-b3c9-3e27ce2b1066 2025-07-31 04:22:50,804 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/4cea2f01-fb49-4c7f-9e2e-05610ea766a1 2025-07-31 04:22:50,835 - INFO - Verified VN network id 11 for VN 4d92304a-da03-4368-b3c9-3e27ce2b1066 2025-07-31 04:22:50,836 - INFO - Verifications in API Server for VN ctest-mgmt_vn-64400332 passed 2025-07-31 04:22:50,836 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/4d92304a-da03-4368-b3c9-3e27ce2b1066 2025-07-31 04:22:50,860 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/4cea2f01-fb49-4c7f-9e2e-05610ea766a1 2025-07-31 04:22:50,875 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/4cea2f01-fb49-4c7f-9e2e-05610ea766a1 2025-07-31 04:22:50,891 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/6fcf8a1e-49f9-4b4c-b7e8-e2a9ef40b02a 2025-07-31 04:22:50,919 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-64400332', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5589583185563566952', 'uuid-lslong': '12954954143791517798'}, 'enable': 'true', 'created': '2025-07-31T04:22:49', 'last-modified': '2025-07-31T04:22:49', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.963841'} 2025-07-31 04:22:50,931 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 04:22:50,941 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-64400332', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5589583185563566952', 'uuid-lslong': '12954954143791517798'}, 'enable': 'true', 'created': '2025-07-31T04:22:49', 'last-modified': '2025-07-31T04:22:49', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.990101'} 2025-07-31 04:22:50,954 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 04:22:50,963 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-64400332', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5589583185563566952', 'uuid-lslong': '12954954143791517798'}, 'enable': 'true', 'created': '2025-07-31T04:22:49', 'last-modified': '2025-07-31T04:22:49', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:01.009581'} 2025-07-31 04:22:50,973 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 04:22:50,973 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-mgmt_vn-64400332 passed 2025-07-31 04:22:50,973 - DEBUG - ====Verifying policy data for ctest-mgmt_vn-64400332 in API_Server ====== 2025-07-31 04:22:50,973 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:22:50,989 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:22:51,024 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:22:51,153 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/4d92304a-da03-4368-b3c9-3e27ce2b1066 2025-07-31 04:22:51,173 - DEBUG - =>VN ctest-mgmt_vn-64400332 has no policy to be verified 2025-07-31 04:22:51,173 - DEBUG - Verifying the vn in opserver 2025-07-31 04:22:51,173 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-64400332 virtual network link through opserver 10.0.0.41 2025-07-31 04:22:51,173 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-07-31 04:22:51,481 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-07-31 04:22:51,602 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-64400332', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-64400332?flat'} 2025-07-31 04:22:51,602 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-64400332 virtual network link through opserver 10.0.0.38 2025-07-31 04:22:51,603 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-07-31 04:22:51,903 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-07-31 04:22:51,973 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-64400332', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-64400332?flat'} 2025-07-31 04:22:51,974 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-64400332 virtual network link through opserver 10.0.0.61 2025-07-31 04:22:51,974 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-07-31 04:22:52,274 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-07-31 04:22:52,375 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-64400332', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-64400332?flat'} 2025-07-31 04:22:52,375 - INFO - Validated that VN default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-64400332 is found in opserver 2025-07-31 04:22:52,429 - DEBUG - Do not have enough data to verify VN in agent 2025-07-31 04:22:52,459 - DEBUG - VRF ids for VN ctest-mgmt_vn-64400332: {} 2025-07-31 04:22:52,799 - DEBUG - Response for create_network : {'network': {'id': 'a25be993-4a0a-40a0-8dcb-48c4ce1b0268', 'name': 'ctest-left_vn-12644923', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'project_id': '5d8e29c3f9ed4135910480d12bc16305', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcHC-45474793', 'ctest-left_vn-12644923'], 'port_security_enabled': True, 'description': ''}} 2025-07-31 04:22:53,032 - DEBUG - Response for create_subnet : {'subnet': {'id': '8da66bec-c402-4582-90f0-30d395f5cb69', 'name': '', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'network_id': 'a25be993-4a0a-40a0-8dcb-48c4ce1b0268', 'ip_version': 4, 'cidr': '5.81.142.192/26', 'allocation_pools': [{'start': '5.81.142.194', 'end': '5.81.142.254'}], 'gateway_ip': '5.81.142.193', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '5.81.142.194', 'tags': [], 'project_id': '5d8e29c3f9ed4135910480d12bc16305'}} 2025-07-31 04:22:53,054 - INFO - Created VN ctest-left_vn-12644923 2025-07-31 04:22:53,069 - DEBUG - VN ctest-left_vn-12644923 UUID is a25be993-4a0a-40a0-8dcb-48c4ce1b0268 2025-07-31 04:22:53,157 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:22:53,166 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:22:53,182 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:22:53,299 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/a25be993-4a0a-40a0-8dcb-48c4ce1b0268 2025-07-31 04:22:53,315 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/a25be993-4a0a-40a0-8dcb-48c4ce1b0268 2025-07-31 04:22:53,330 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/2f3ddedd-e584-4eca-9ee7-0edaaa105948 2025-07-31 04:22:53,344 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/2f3ddedd-e584-4eca-9ee7-0edaaa105948 2025-07-31 04:22:53,354 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/b9a5199a-ff7a-4cd1-bcd3-4f6cf6a623d5 2025-07-31 04:22:53,363 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-07-31 04:22:53,363 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/a25be993-4a0a-40a0-8dcb-48c4ce1b0268 2025-07-31 04:22:53,377 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/2f3ddedd-e584-4eca-9ee7-0edaaa105948 2025-07-31 04:22:53,394 - INFO - Verified VN network id 12 for VN a25be993-4a0a-40a0-8dcb-48c4ce1b0268 2025-07-31 04:22:53,395 - INFO - Verifications in API Server for VN ctest-left_vn-12644923 passed 2025-07-31 04:22:53,395 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/a25be993-4a0a-40a0-8dcb-48c4ce1b0268 2025-07-31 04:22:53,410 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/2f3ddedd-e584-4eca-9ee7-0edaaa105948 2025-07-31 04:22:53,421 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/2f3ddedd-e584-4eca-9ee7-0edaaa105948 2025-07-31 04:22:53,431 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/b9a5199a-ff7a-4cd1-bcd3-4f6cf6a623d5 2025-07-31 04:22:53,447 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-12644923', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '11699201275836645536', 'uuid-lslong': '10217340189728768616'}, 'enable': 'true', 'created': '2025-07-31T04:22:52', 'last-modified': '2025-07-31T04:22:52', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.486393'} 2025-07-31 04:22:53,456 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-07-31 04:22:53,464 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-12644923', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '11699201275836645536', 'uuid-lslong': '10217340189728768616'}, 'enable': 'true', 'created': '2025-07-31T04:22:52', 'last-modified': '2025-07-31T04:22:52', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.499912'} 2025-07-31 04:22:53,471 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-07-31 04:22:53,478 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-12644923', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '11699201275836645536', 'uuid-lslong': '10217340189728768616'}, 'enable': 'true', 'created': '2025-07-31T04:22:52', 'last-modified': '2025-07-31T04:22:52', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.513696'} 2025-07-31 04:22:53,486 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-07-31 04:22:53,486 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-left_vn-12644923 passed 2025-07-31 04:22:53,486 - DEBUG - ====Verifying policy data for ctest-left_vn-12644923 in API_Server ====== 2025-07-31 04:22:53,486 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:22:53,496 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:22:53,513 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:22:53,651 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/a25be993-4a0a-40a0-8dcb-48c4ce1b0268 2025-07-31 04:22:53,668 - DEBUG - =>VN ctest-left_vn-12644923 has no policy to be verified 2025-07-31 04:22:53,668 - DEBUG - Verifying the vn in opserver 2025-07-31 04:22:53,668 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-12644923 virtual network link through opserver 10.0.0.41 2025-07-31 04:22:53,668 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-07-31 04:22:53,744 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-12644923', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-12644923?flat'} 2025-07-31 04:22:53,745 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-12644923 virtual network link through opserver 10.0.0.38 2025-07-31 04:22:53,745 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-07-31 04:22:53,816 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-12644923', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-12644923?flat'} 2025-07-31 04:22:53,816 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-12644923 virtual network link through opserver 10.0.0.61 2025-07-31 04:22:53,816 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-07-31 04:22:53,886 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-12644923', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-12644923?flat'} 2025-07-31 04:22:53,886 - INFO - Validated that VN default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-12644923 is found in opserver 2025-07-31 04:22:53,927 - DEBUG - Do not have enough data to verify VN in agent 2025-07-31 04:22:53,951 - DEBUG - VRF ids for VN ctest-left_vn-12644923: {} 2025-07-31 04:22:54,327 - DEBUG - Response for create_network : {'network': {'id': '0b7df74d-9233-4fec-bed8-989236cad5d4', 'name': 'ctest-right_vn-29490384', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'project_id': '5d8e29c3f9ed4135910480d12bc16305', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcHC-45474793', 'ctest-right_vn-29490384'], 'port_security_enabled': True, 'description': ''}} 2025-07-31 04:22:54,613 - DEBUG - Response for create_subnet : {'subnet': {'id': '96e25999-d17a-42b2-9adb-995a68350278', 'name': '', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'network_id': '0b7df74d-9233-4fec-bed8-989236cad5d4', 'ip_version': 4, 'cidr': '195.236.99.0/26', 'allocation_pools': [{'start': '195.236.99.2', 'end': '195.236.99.62'}], 'gateway_ip': '195.236.99.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '195.236.99.2', 'tags': [], 'project_id': '5d8e29c3f9ed4135910480d12bc16305'}} 2025-07-31 04:22:54,638 - INFO - Created VN ctest-right_vn-29490384 2025-07-31 04:22:54,653 - DEBUG - VN ctest-right_vn-29490384 UUID is 0b7df74d-9233-4fec-bed8-989236cad5d4 2025-07-31 04:22:54,734 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:22:54,743 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:22:54,761 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:22:54,870 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/0b7df74d-9233-4fec-bed8-989236cad5d4 2025-07-31 04:22:54,893 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/0b7df74d-9233-4fec-bed8-989236cad5d4 2025-07-31 04:22:54,908 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/044d05ce-71ef-4ad0-8339-73cecacd563b 2025-07-31 04:22:54,919 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/044d05ce-71ef-4ad0-8339-73cecacd563b 2025-07-31 04:22:54,935 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/a2c563ca-b77e-4164-a132-c879280bbe31 2025-07-31 04:22:54,955 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-07-31 04:22:54,955 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/0b7df74d-9233-4fec-bed8-989236cad5d4 2025-07-31 04:22:54,976 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/044d05ce-71ef-4ad0-8339-73cecacd563b 2025-07-31 04:22:55,003 - INFO - Verified VN network id 13 for VN 0b7df74d-9233-4fec-bed8-989236cad5d4 2025-07-31 04:22:55,004 - INFO - Verifications in API Server for VN ctest-right_vn-29490384 passed 2025-07-31 04:22:55,004 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/0b7df74d-9233-4fec-bed8-989236cad5d4 2025-07-31 04:22:55,038 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/044d05ce-71ef-4ad0-8339-73cecacd563b 2025-07-31 04:22:55,053 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/044d05ce-71ef-4ad0-8339-73cecacd563b 2025-07-31 04:22:55,067 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/a2c563ca-b77e-4164-a132-c879280bbe31 2025-07-31 04:22:55,092 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-29490384', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '828089819043418092', 'uuid-lslong': '13751909215927719380'}, 'enable': 'true', 'created': '2025-07-31T04:22:54', 'last-modified': '2025-07-31T04:22:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.546677'} 2025-07-31 04:22:55,103 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-07-31 04:22:55,112 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-29490384', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '828089819043418092', 'uuid-lslong': '13751909215927719380'}, 'enable': 'true', 'created': '2025-07-31T04:22:54', 'last-modified': '2025-07-31T04:22:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.566635'} 2025-07-31 04:22:55,122 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-07-31 04:22:55,132 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-29490384', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '828089819043418092', 'uuid-lslong': '13751909215927719380'}, 'enable': 'true', 'created': '2025-07-31T04:22:54', 'last-modified': '2025-07-31T04:22:54', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.584279'} 2025-07-31 04:22:55,141 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-07-31 04:22:55,142 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-right_vn-29490384 passed 2025-07-31 04:22:55,142 - DEBUG - ====Verifying policy data for ctest-right_vn-29490384 in API_Server ====== 2025-07-31 04:22:55,142 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:22:55,157 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:22:55,197 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:22:55,345 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/0b7df74d-9233-4fec-bed8-989236cad5d4 2025-07-31 04:22:55,369 - DEBUG - =>VN ctest-right_vn-29490384 has no policy to be verified 2025-07-31 04:22:55,369 - DEBUG - Verifying the vn in opserver 2025-07-31 04:22:55,370 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-29490384 virtual network link through opserver 10.0.0.41 2025-07-31 04:22:55,370 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-07-31 04:22:55,455 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-29490384', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-29490384?flat'} 2025-07-31 04:22:55,456 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-29490384 virtual network link through opserver 10.0.0.38 2025-07-31 04:22:55,456 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-07-31 04:22:55,568 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-29490384', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-29490384?flat'} 2025-07-31 04:22:55,568 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-29490384 virtual network link through opserver 10.0.0.61 2025-07-31 04:22:55,568 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-07-31 04:22:55,698 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-29490384', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-29490384?flat'} 2025-07-31 04:22:55,698 - INFO - Validated that VN default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-29490384 is found in opserver 2025-07-31 04:22:55,745 - DEBUG - Do not have enough data to verify VN in agent 2025-07-31 04:22:55,769 - DEBUG - VRF ids for VN ctest-right_vn-29490384: {} 2025-07-31 04:22:55,897 - DEBUG - Services list from nova: [, , , ] 2025-07-31 04:22:57,579 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-1) 2025-07-31 04:22:59,109 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-2) 2025-07-31 04:22:59,109 - DEBUG - Creating service template: ['default-domain', 'ctest-service_template_1-58265318'] 2025-07-31 04:22:59,424 - DEBUG - Service template: ['default-domain', 'ctest-service_template_1-58265318'] created succesfully 2025-07-31 04:24:40,494 - DEBUG - Installing image tiny_nat_fw 2025-07-31 04:24:40,494 - DEBUG - Image info {'name': 'tinycore-in-network-nat.qcow2.gz', 'webserver': '', 'location': '/images/tinycore/', 'username': 'tc', 'password': 'secret', 'type': 'os', 'flavor': 'contrail_flavor_tiny', 'params': '--container-format bare --disk-format vmdk --property vmware_disktype="sparse" --property vmware_adaptertype="ide" --property hypervisor_type=qemu', 'name_docker': 'phusion-baseimage-enablesshd', 'vctmpl': 'service_chain_images/nat_service/tiny_nat_fw/tinycore-in-network-nat.vmtx', 'vcname': 'service_chain_images/nat_service/tiny_nat_fw/tinycore-in-network-nat.vmdk', 'vmdk': 'service_chain_images/nat_service/tiny_nat_fw/tinycore-in-network-nat-flat.vmdk', 'shrinked_vmdk': "'no'"} 2025-07-31 04:24:40,495 - DEBUG - Image build path file:///contrail-test/images/tinycore-in-network-nat.qcow2.gz 2025-07-31 04:24:40,495 - DEBUG - Download image from local file file:///contrail-test/images/tinycore-in-network-nat.qcow2.gz 2025-07-31 04:24:40,495 - DEBUG - Unzip image 2025-07-31 04:24:40,495 - DEBUG - Image download (local=True) (proxy=None): Execute cmd: gunzip -f /contrail-test/images/tinycore-in-network-nat.qcow2.gz 2025-07-31 04:24:40,536 - DEBUG - Try to create image in user context 2025-07-31 04:24:40,647 - DEBUG - Deleting service template: ['default-domain', 'ctest-service_template_1-58265318'] 2025-07-31 04:24:40,757 - INFO - Service template: ['default-domain', 'ctest-service_template_1-58265318'] deleted successfully. 2025-07-31 04:24:40,757 - INFO - Skip interface_detach for VM ctest-right_vm-81200336 in state BUILD 2025-07-31 04:24:40,757 - INFO - Deleting VM ctest-right_vm-81200336 2025-07-31 04:24:40,789 - ERROR - VM ctest-right_vm-81200336 has failed to come up 2025-07-31 04:24:40,789 - ERROR - Nova failed to get host of the VM 2025-07-31 04:24:40,790 - ERROR - VM ctest-right_vm-81200336 has failed to come up 2025-07-31 04:24:40,790 - ERROR - Nova failed to get host of the VM 2025-07-31 04:24:40,870 - INFO - Skip interface_detach for VM ctest-left_vm-33336629 in state BUILD 2025-07-31 04:24:40,871 - INFO - Deleting VM ctest-left_vm-33336629 2025-07-31 04:24:40,871 - ERROR - VM ctest-left_vm-33336629 has failed to come up 2025-07-31 04:24:40,871 - ERROR - Nova failed to get host of the VM 2025-07-31 04:24:40,871 - ERROR - VM ctest-left_vm-33336629 has failed to come up 2025-07-31 04:24:40,871 - ERROR - Nova failed to get host of the VM 2025-07-31 04:24:40,988 - INFO - Deleting VN ctest-right_vn-29490384 2025-07-31 04:24:41,063 - DEBUG - VN 0b7df74d-9233-4fec-bed8-989236cad5d4 still in use: Unable to complete operation on network 0b7df74d-9233-4fec-bed8-989236cad5d4. There are one or more ports still in use on the network. Neutron server returns request_ids: ['req-7f4d5eb7-bb96-4d00-8735-558955d93ff4'] 2025-07-31 04:24:41,064 - WARNING - Deleting VN ctest-right_vn-29490384 failed..Will retry 2025-07-31 04:24:43,270 - DEBUG - Response for deleting network () 2025-07-31 04:24:43,271 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/044d05ce-71ef-4ad0-8339-73cecacd563b 2025-07-31 04:24:43,278 - DEBUG - Response Code: 404 2025-07-31 04:24:43,278 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:24:43,286 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:24:43,315 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:24:43,420 - INFO - Validated that VN ctest-right_vn-29490384 is not found in API Server 2025-07-31 04:24:43,437 - DEBUG - VN ctest-right_vn-29490384 is not present in Agent 10.0.0.27 2025-07-31 04:24:43,451 - DEBUG - VN ctest-right_vn-29490384 is not present in Agent 10.0.0.57 2025-07-31 04:24:43,451 - INFO - Validated that VN ctest-right_vn-29490384 is not in any agent 2025-07-31 04:24:43,502 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-right_vn-29490384 info 2025-07-31 04:24:43,503 - INFO - Deleting VN ctest-left_vn-12644923 2025-07-31 04:24:43,693 - DEBUG - Response for deleting network () 2025-07-31 04:24:43,693 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/2f3ddedd-e584-4eca-9ee7-0edaaa105948 2025-07-31 04:24:43,699 - DEBUG - Response Code: 404 2025-07-31 04:24:43,699 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:24:43,715 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:24:43,740 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:24:43,847 - INFO - Validated that VN ctest-left_vn-12644923 is not found in API Server 2025-07-31 04:24:43,862 - DEBUG - VN ctest-left_vn-12644923 is not present in Agent 10.0.0.27 2025-07-31 04:24:43,875 - DEBUG - VN ctest-left_vn-12644923 is not present in Agent 10.0.0.57 2025-07-31 04:24:43,876 - INFO - Validated that VN ctest-left_vn-12644923 is not in any agent 2025-07-31 04:24:43,921 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-left_vn-12644923 info 2025-07-31 04:24:43,922 - INFO - Deleting VN ctest-mgmt_vn-64400332 2025-07-31 04:24:44,102 - DEBUG - Response for deleting network () 2025-07-31 04:24:44,102 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/4cea2f01-fb49-4c7f-9e2e-05610ea766a1 2025-07-31 04:24:44,114 - DEBUG - Response Code: 404 2025-07-31 04:24:44,114 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:24:44,122 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:24:44,183 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:24:44,282 - INFO - Validated that VN ctest-mgmt_vn-64400332 is not found in API Server 2025-07-31 04:24:44,297 - DEBUG - VN ctest-mgmt_vn-64400332 is not present in Agent 10.0.0.27 2025-07-31 04:24:44,317 - DEBUG - VN ctest-mgmt_vn-64400332 is not present in Agent 10.0.0.57 2025-07-31 04:24:44,317 - INFO - Validated that VN ctest-mgmt_vn-64400332 is not in any agent 2025-07-31 04:24:44,368 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-mgmt_vn-64400332 info 2025-07-31 04:24:45,668 - ERROR - 
Traceback (most recent call last):
  File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
    result = function(self, *args, **kwargs)
  File "/contrail-test/scripts/svc_health_check/test_svc_health_check.py", line 16, in test_svc_hc_basic
    create_svms=True, max_inst=1)
  File "/contrail-test/common/servicechain/verify.py", line 320, in verify_svc_chain
    ret_dict = svc_chain_info or self.config_svc_chain(*args, **kwargs)
  File "/contrail-test/common/servicechain/config.py", line 533, in config_svc_chain
    max_inst=max_inst)
  File "/contrail-test/common/servicechain/config.py", line 335, in create_service_vms
    zone=random.choice(valid_zones))
  File "/contrail-test/common/servicechain/config.py", line 161, in config_vm_only
    flavor=flavor, zone=zone, node_name=node_name,**kwargs)
  File "/contrail-test/common/servicechain/config.py", line 203, in config_vm
    flavor=flavor, vn_objs=vns, count=count, zone=zone,**kwargs))
  File "/usr/local/lib/python3.6/site-packages/testtools/testcase.py", line 756, in useFixture
    reraise(*exc_info)
  File "/usr/local/lib/python3.6/site-packages/testtools/_compat3x.py", line 16, in reraise
    raise exc_obj.with_traceback(exc_tb)
  File "/usr/local/lib/python3.6/site-packages/testtools/testcase.py", line 731, in useFixture
    fixture.setUp()
  File "/contrail-test/fixtures/vm_test.py", line 188, in setUp
    self.create()
  File "/contrail-test/fixtures/vm_test.py", line 240, in create
    fixed_ips=self.fixed_ips)
  File "/contrail-test/fixtures/openstack.py", line 118, in create_vm
    return self.nova_h.create_vm(vm_name=vm_name, image_name=image_name, vn_ids=vn_ids, **kwargs)
  File "/contrail-test/fixtures/nova_test.py", line 577, in create_vm
    image = self.get_image(image_name=image_name)
  File "/contrail-test/fixtures/nova_test.py", line 198, in get_image
    self._install_image(image_name=image_name)
  File "/contrail-test/fixtures/nova_test.py", line 338, in _install_image
    return self.copy_and_glance(build_path, image_name, params)
  File "/contrail-test/fixtures/nova_test.py", line 407, in copy_and_glance
    image_id = self.glance_h.create_image(generic_image_name, image_path_real, **params)
  File "/contrail-test/fixtures/glance_test.py", line 50, in create_image
    self.upload_image(obj['id'], filename)
  File "/contrail-test/fixtures/glance_test.py", line 54, in upload_image
    self.obj.images.upload(uuid, open(filename, 'rb'))
FileNotFoundError: [Errno 2] No such file or directory: '/contrail-test/images/tinycore-in-network-nat.qcow2'
2025-07-31 04:24:45,688 - DEBUG - No XMPP flaps were noticed during the test 2025-07-31 04:24:45,689 - INFO - 2025-07-31 04:24:45,689 - INFO - END TEST : test_svc_hc_basic : FAILED[0:01:58] 2025-07-31 04:24:45,689 - INFO - -------------------------------------------------------------------------------- 2025-07-31 04:24:45,702 - INFO - ================================================================================ 2025-07-31 04:24:45,702 - INFO - STARTING TEST : test_svc_trans_segment_right_hc_fail_svm 2025-07-31 04:24:47,045 - DEBUG - Nothing to compare xmpp stats {'10.0.0.27': {'10.20.0.193': '0', '10.20.0.18': '0'}, '10.0.0.57': {'10.20.0.18': '0', '10.20.0.193': '0'}} with 2025-07-31 04:24:47,046 - INFO - Initial checks done. Running the testcase now 2025-07-31 04:24:47,046 - INFO - 2025-07-31 04:24:47,339 - DEBUG - Response for create_network : {'network': {'id': '937f5da2-50f6-4308-9327-3618eebe61e6', 'name': 'ctest-mgmt_vn-81093085', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'project_id': '5d8e29c3f9ed4135910480d12bc16305', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcHC-45474793', 'ctest-mgmt_vn-81093085'], 'port_security_enabled': True, 'description': ''}} 2025-07-31 04:24:47,633 - DEBUG - Response for create_subnet : {'subnet': {'id': '8113dd48-1236-4969-b18b-fb6d86dea888', 'name': '', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'network_id': '937f5da2-50f6-4308-9327-3618eebe61e6', 'ip_version': 4, 'cidr': '46.7.53.0/26', 'allocation_pools': [{'start': '46.7.53.2', 'end': '46.7.53.62'}], 'gateway_ip': '46.7.53.1', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '46.7.53.2', 'tags': [], 'project_id': '5d8e29c3f9ed4135910480d12bc16305'}} 2025-07-31 04:24:47,662 - INFO - Created VN ctest-mgmt_vn-81093085 2025-07-31 04:24:47,680 - DEBUG - VN ctest-mgmt_vn-81093085 UUID is 937f5da2-50f6-4308-9327-3618eebe61e6 2025-07-31 04:24:47,760 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:24:47,767 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:24:47,781 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:24:47,889 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/937f5da2-50f6-4308-9327-3618eebe61e6 2025-07-31 04:24:47,911 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/937f5da2-50f6-4308-9327-3618eebe61e6 2025-07-31 04:24:47,923 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/371c6b36-9bdd-4a69-91de-c6c7fc415960 2025-07-31 04:24:47,932 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/371c6b36-9bdd-4a69-91de-c6c7fc415960 2025-07-31 04:24:47,944 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/537e8320-09c3-43c8-9b4e-6a64097ae3f2 2025-07-31 04:24:47,953 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 04:24:47,953 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/937f5da2-50f6-4308-9327-3618eebe61e6 2025-07-31 04:24:47,968 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/371c6b36-9bdd-4a69-91de-c6c7fc415960 2025-07-31 04:24:47,991 - INFO - Verified VN network id 11 for VN 937f5da2-50f6-4308-9327-3618eebe61e6 2025-07-31 04:24:47,991 - INFO - Verifications in API Server for VN ctest-mgmt_vn-81093085 passed 2025-07-31 04:24:47,991 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/937f5da2-50f6-4308-9327-3618eebe61e6 2025-07-31 04:24:48,005 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/371c6b36-9bdd-4a69-91de-c6c7fc415960 2025-07-31 04:24:48,017 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/371c6b36-9bdd-4a69-91de-c6c7fc415960 2025-07-31 04:24:48,031 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/537e8320-09c3-43c8-9b4e-6a64097ae3f2 2025-07-31 04:24:48,049 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-81093085', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '10628316597342061320', 'uuid-lslong': '10603503328379691494'}, 'enable': 'true', 'created': '2025-07-31T04:24:47', 'last-modified': '2025-07-31T04:24:47', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.479278'} 2025-07-31 04:24:48,061 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 04:24:48,069 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-81093085', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '10628316597342061320', 'uuid-lslong': '10603503328379691494'}, 'enable': 'true', 'created': '2025-07-31T04:24:47', 'last-modified': '2025-07-31T04:24:47', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.498395'} 2025-07-31 04:24:48,077 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 04:24:48,083 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-81093085', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '10628316597342061320', 'uuid-lslong': '10603503328379691494'}, 'enable': 'true', 'created': '2025-07-31T04:24:47', 'last-modified': '2025-07-31T04:24:47', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.512014'} 2025-07-31 04:24:48,091 - DEBUG - Route Targets: ['target:64512:8000005'] 2025-07-31 04:24:48,091 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-mgmt_vn-81093085 passed 2025-07-31 04:24:48,091 - DEBUG - ====Verifying policy data for ctest-mgmt_vn-81093085 in API_Server ====== 2025-07-31 04:24:48,091 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:24:48,100 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:24:48,119 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:24:48,225 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/937f5da2-50f6-4308-9327-3618eebe61e6 2025-07-31 04:24:48,243 - DEBUG - =>VN ctest-mgmt_vn-81093085 has no policy to be verified 2025-07-31 04:24:48,243 - DEBUG - Verifying the vn in opserver 2025-07-31 04:24:48,243 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-81093085 virtual network link through opserver 10.0.0.41 2025-07-31 04:24:48,243 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-07-31 04:24:48,258 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-81093085', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-81093085?flat'} 2025-07-31 04:24:48,258 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-81093085 virtual network link through opserver 10.0.0.38 2025-07-31 04:24:48,258 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-07-31 04:24:48,275 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-81093085', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-81093085?flat'} 2025-07-31 04:24:48,276 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-81093085 virtual network link through opserver 10.0.0.61 2025-07-31 04:24:48,276 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-07-31 04:24:48,291 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-81093085', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-81093085?flat'} 2025-07-31 04:24:48,291 - INFO - Validated that VN default-domain:ctest-TestSvcHC-45474793:ctest-mgmt_vn-81093085 is found in opserver 2025-07-31 04:24:48,319 - DEBUG - Do not have enough data to verify VN in agent 2025-07-31 04:24:48,334 - DEBUG - VRF ids for VN ctest-mgmt_vn-81093085: {} 2025-07-31 04:24:48,612 - DEBUG - Response for create_network : {'network': {'id': '05751701-20fc-4418-879d-e5a1b873ac6d', 'name': 'ctest-left_vn-82355333', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'project_id': '5d8e29c3f9ed4135910480d12bc16305', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcHC-45474793', 'ctest-left_vn-82355333'], 'port_security_enabled': True, 'description': ''}} 2025-07-31 04:24:48,882 - DEBUG - Response for create_subnet : {'subnet': {'id': '6825f038-de72-4dd1-a2d5-cd5137a62a4a', 'name': '', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'network_id': '05751701-20fc-4418-879d-e5a1b873ac6d', 'ip_version': 4, 'cidr': '102.201.118.128/26', 'allocation_pools': [{'start': '102.201.118.130', 'end': '102.201.118.190'}], 'gateway_ip': '102.201.118.129', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '102.201.118.130', 'tags': [], 'project_id': '5d8e29c3f9ed4135910480d12bc16305'}} 2025-07-31 04:24:48,906 - INFO - Created VN ctest-left_vn-82355333 2025-07-31 04:24:48,919 - DEBUG - VN ctest-left_vn-82355333 UUID is 05751701-20fc-4418-879d-e5a1b873ac6d 2025-07-31 04:24:49,010 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:24:49,019 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:24:49,038 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:24:49,156 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/05751701-20fc-4418-879d-e5a1b873ac6d 2025-07-31 04:24:49,172 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/05751701-20fc-4418-879d-e5a1b873ac6d 2025-07-31 04:24:49,186 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/aa2c0b6c-49f3-4fa2-a9b4-f881ff5c5a5b 2025-07-31 04:24:49,199 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/aa2c0b6c-49f3-4fa2-a9b4-f881ff5c5a5b 2025-07-31 04:24:49,213 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/7c603ee6-fd72-4ac9-9905-f82b8df1ccb6 2025-07-31 04:24:49,225 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-07-31 04:24:49,225 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/05751701-20fc-4418-879d-e5a1b873ac6d 2025-07-31 04:24:49,239 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/aa2c0b6c-49f3-4fa2-a9b4-f881ff5c5a5b 2025-07-31 04:24:49,257 - INFO - Verified VN network id 12 for VN 05751701-20fc-4418-879d-e5a1b873ac6d 2025-07-31 04:24:49,257 - INFO - Verifications in API Server for VN ctest-left_vn-82355333 passed 2025-07-31 04:24:49,257 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/05751701-20fc-4418-879d-e5a1b873ac6d 2025-07-31 04:24:49,279 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/aa2c0b6c-49f3-4fa2-a9b4-f881ff5c5a5b 2025-07-31 04:24:49,296 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/aa2c0b6c-49f3-4fa2-a9b4-f881ff5c5a5b 2025-07-31 04:24:49,312 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/7c603ee6-fd72-4ac9-9905-f82b8df1ccb6 2025-07-31 04:24:49,331 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-82355333', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '393245836080595992', 'uuid-lslong': '9772219249210928237'}, 'enable': 'true', 'created': '2025-07-31T04:24:48', 'last-modified': '2025-07-31T04:24:48', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.510909'} 2025-07-31 04:24:49,339 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-07-31 04:24:49,347 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-82355333', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '393245836080595992', 'uuid-lslong': '9772219249210928237'}, 'enable': 'true', 'created': '2025-07-31T04:24:48', 'last-modified': '2025-07-31T04:24:48', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.526528'} 2025-07-31 04:24:49,354 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-07-31 04:24:49,361 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-82355333', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '393245836080595992', 'uuid-lslong': '9772219249210928237'}, 'enable': 'true', 'created': '2025-07-31T04:24:48', 'last-modified': '2025-07-31T04:24:48', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.538255'} 2025-07-31 04:24:49,369 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-07-31 04:24:49,370 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-left_vn-82355333 passed 2025-07-31 04:24:49,370 - DEBUG - ====Verifying policy data for ctest-left_vn-82355333 in API_Server ====== 2025-07-31 04:24:49,370 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:24:49,381 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:24:49,411 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:24:49,542 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/05751701-20fc-4418-879d-e5a1b873ac6d 2025-07-31 04:24:49,565 - DEBUG - =>VN ctest-left_vn-82355333 has no policy to be verified 2025-07-31 04:24:49,565 - DEBUG - Verifying the vn in opserver 2025-07-31 04:24:49,565 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-82355333 virtual network link through opserver 10.0.0.41 2025-07-31 04:24:49,565 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-07-31 04:24:49,587 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-82355333', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-82355333?flat'} 2025-07-31 04:24:49,587 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-82355333 virtual network link through opserver 10.0.0.38 2025-07-31 04:24:49,587 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-07-31 04:24:49,610 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-82355333', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-82355333?flat'} 2025-07-31 04:24:49,610 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-82355333 virtual network link through opserver 10.0.0.61 2025-07-31 04:24:49,611 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-07-31 04:24:49,629 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-82355333', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-82355333?flat'} 2025-07-31 04:24:49,629 - INFO - Validated that VN default-domain:ctest-TestSvcHC-45474793:ctest-left_vn-82355333 is found in opserver 2025-07-31 04:24:49,676 - DEBUG - Do not have enough data to verify VN in agent 2025-07-31 04:24:49,702 - DEBUG - VRF ids for VN ctest-left_vn-82355333: {} 2025-07-31 04:24:49,981 - DEBUG - Response for create_network : {'network': {'id': '16509fe8-0b56-4f75-81d0-a95ce9197f81', 'name': 'ctest-right_vn-71731210', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'project_id': '5d8e29c3f9ed4135910480d12bc16305', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcHC-45474793', 'ctest-right_vn-71731210'], 'port_security_enabled': True, 'description': ''}} 2025-07-31 04:24:50,224 - DEBUG - Response for create_subnet : {'subnet': {'id': 'a652fe88-a52e-4bf7-a0bc-2586f12b7813', 'name': '', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'network_id': '16509fe8-0b56-4f75-81d0-a95ce9197f81', 'ip_version': 4, 'cidr': '157.47.117.128/26', 'allocation_pools': [{'start': '157.47.117.130', 'end': '157.47.117.190'}], 'gateway_ip': '157.47.117.129', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '157.47.117.130', 'tags': [], 'project_id': '5d8e29c3f9ed4135910480d12bc16305'}} 2025-07-31 04:24:50,246 - INFO - Created VN ctest-right_vn-71731210 2025-07-31 04:24:50,259 - DEBUG - VN ctest-right_vn-71731210 UUID is 16509fe8-0b56-4f75-81d0-a95ce9197f81 2025-07-31 04:24:50,346 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:24:50,355 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:24:50,372 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:24:50,480 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/16509fe8-0b56-4f75-81d0-a95ce9197f81 2025-07-31 04:24:50,498 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/16509fe8-0b56-4f75-81d0-a95ce9197f81 2025-07-31 04:24:50,511 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/e6a1f278-a0dc-490b-9ee0-44190d508ed3 2025-07-31 04:24:50,523 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/e6a1f278-a0dc-490b-9ee0-44190d508ed3 2025-07-31 04:24:50,535 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/fc67b2e1-6268-4172-9d13-1310c0000369 2025-07-31 04:24:50,546 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-07-31 04:24:50,546 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/16509fe8-0b56-4f75-81d0-a95ce9197f81 2025-07-31 04:24:50,563 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/e6a1f278-a0dc-490b-9ee0-44190d508ed3 2025-07-31 04:24:50,586 - INFO - Verified VN network id 13 for VN 16509fe8-0b56-4f75-81d0-a95ce9197f81 2025-07-31 04:24:50,586 - INFO - Verifications in API Server for VN ctest-right_vn-71731210 passed 2025-07-31 04:24:50,586 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/16509fe8-0b56-4f75-81d0-a95ce9197f81 2025-07-31 04:24:50,602 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/e6a1f278-a0dc-490b-9ee0-44190d508ed3 2025-07-31 04:24:50,613 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/e6a1f278-a0dc-490b-9ee0-44190d508ed3 2025-07-31 04:24:50,624 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/fc67b2e1-6268-4172-9d13-1310c0000369 2025-07-31 04:24:50,648 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-71731210', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1607960885942701941', 'uuid-lslong': '9354162642561367937'}, 'enable': 'true', 'created': '2025-07-31T04:24:49', 'last-modified': '2025-07-31T04:24:50', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.483896'} 2025-07-31 04:24:50,658 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-07-31 04:24:50,668 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-71731210', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1607960885942701941', 'uuid-lslong': '9354162642561367937'}, 'enable': 'true', 'created': '2025-07-31T04:24:49', 'last-modified': '2025-07-31T04:24:50', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.505798'} 2025-07-31 04:24:50,679 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-07-31 04:24:50,690 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-71731210', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1607960885942701941', 'uuid-lslong': '9354162642561367937'}, 'enable': 'true', 'created': '2025-07-31T04:24:49', 'last-modified': '2025-07-31T04:24:50', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.520413'} 2025-07-31 04:24:50,701 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-07-31 04:24:50,701 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-right_vn-71731210 passed 2025-07-31 04:24:50,701 - DEBUG - ====Verifying policy data for ctest-right_vn-71731210 in API_Server ====== 2025-07-31 04:24:50,701 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:24:50,714 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:24:50,739 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:24:50,912 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/16509fe8-0b56-4f75-81d0-a95ce9197f81 2025-07-31 04:24:50,937 - DEBUG - =>VN ctest-right_vn-71731210 has no policy to be verified 2025-07-31 04:24:50,937 - DEBUG - Verifying the vn in opserver 2025-07-31 04:24:50,937 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-71731210 virtual network link through opserver 10.0.0.41 2025-07-31 04:24:50,937 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-07-31 04:24:50,960 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-71731210', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-71731210?flat'} 2025-07-31 04:24:50,960 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-71731210 virtual network link through opserver 10.0.0.38 2025-07-31 04:24:50,960 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-07-31 04:24:50,982 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-71731210', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-71731210?flat'} 2025-07-31 04:24:50,982 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-71731210 virtual network link through opserver 10.0.0.61 2025-07-31 04:24:50,982 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-07-31 04:24:51,003 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-71731210', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-71731210?flat'} 2025-07-31 04:24:51,004 - INFO - Validated that VN default-domain:ctest-TestSvcHC-45474793:ctest-right_vn-71731210 is found in opserver 2025-07-31 04:24:51,040 - DEBUG - Do not have enough data to verify VN in agent 2025-07-31 04:24:51,060 - DEBUG - VRF ids for VN ctest-right_vn-71731210: {} 2025-07-31 04:24:51,342 - DEBUG - Response for create_network : {'network': {'id': '140db3b8-7231-4a9c-ae16-b835272ad34c', 'name': 'ctest-trans_left_vn-34185992', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'project_id': '5d8e29c3f9ed4135910480d12bc16305', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcHC-45474793', 'ctest-trans_left_vn-34185992'], 'port_security_enabled': True, 'description': ''}} 2025-07-31 04:24:51,612 - DEBUG - Response for create_subnet : {'subnet': {'id': '05e28104-2f67-445e-ac99-7ca78964c9ff', 'name': '', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'network_id': '140db3b8-7231-4a9c-ae16-b835272ad34c', 'ip_version': 4, 'cidr': '45.117.236.192/26', 'allocation_pools': [{'start': '45.117.236.194', 'end': '45.117.236.254'}], 'gateway_ip': '45.117.236.193', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '45.117.236.194', 'tags': [], 'project_id': '5d8e29c3f9ed4135910480d12bc16305'}} 2025-07-31 04:24:51,639 - INFO - Created VN ctest-trans_left_vn-34185992 2025-07-31 04:24:51,657 - DEBUG - VN ctest-trans_left_vn-34185992 UUID is 140db3b8-7231-4a9c-ae16-b835272ad34c 2025-07-31 04:24:51,752 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:24:51,764 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:24:51,784 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:24:51,907 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/140db3b8-7231-4a9c-ae16-b835272ad34c 2025-07-31 04:24:51,927 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/140db3b8-7231-4a9c-ae16-b835272ad34c 2025-07-31 04:24:51,943 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/87c33d3b-934a-43a1-9fe8-45ceae2c8310 2025-07-31 04:24:51,955 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/87c33d3b-934a-43a1-9fe8-45ceae2c8310 2025-07-31 04:24:51,968 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/71165ab7-658c-4c8a-b841-4840ffdb63fe 2025-07-31 04:24:51,980 - DEBUG - Route Targets: ['target:64512:8000008'] 2025-07-31 04:24:51,980 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/140db3b8-7231-4a9c-ae16-b835272ad34c 2025-07-31 04:24:51,993 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/87c33d3b-934a-43a1-9fe8-45ceae2c8310 2025-07-31 04:24:52,017 - INFO - Verified VN network id 14 for VN 140db3b8-7231-4a9c-ae16-b835272ad34c 2025-07-31 04:24:52,017 - INFO - Verifications in API Server for VN ctest-trans_left_vn-34185992 passed 2025-07-31 04:24:52,017 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/140db3b8-7231-4a9c-ae16-b835272ad34c 2025-07-31 04:24:52,034 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/87c33d3b-934a-43a1-9fe8-45ceae2c8310 2025-07-31 04:24:52,046 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/87c33d3b-934a-43a1-9fe8-45ceae2c8310 2025-07-31 04:24:52,058 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/71165ab7-658c-4c8a-b841-4840ffdb63fe 2025-07-31 04:24:52,080 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-trans_left_vn-34185992', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1445008660226984604', 'uuid-lslong': '12544416350516990796'}, 'enable': 'true', 'created': '2025-07-31T04:24:51', 'last-modified': '2025-07-31T04:24:51', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.532073'} 2025-07-31 04:24:52,090 - DEBUG - Route Targets: ['target:64512:8000008'] 2025-07-31 04:24:52,099 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-trans_left_vn-34185992', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1445008660226984604', 'uuid-lslong': '12544416350516990796'}, 'enable': 'true', 'created': '2025-07-31T04:24:51', 'last-modified': '2025-07-31T04:24:51', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.553402'} 2025-07-31 04:24:52,110 - DEBUG - Route Targets: ['target:64512:8000008'] 2025-07-31 04:24:52,118 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-trans_left_vn-34185992', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1445008660226984604', 'uuid-lslong': '12544416350516990796'}, 'enable': 'true', 'created': '2025-07-31T04:24:51', 'last-modified': '2025-07-31T04:24:51', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.572771'} 2025-07-31 04:24:52,131 - DEBUG - Route Targets: ['target:64512:8000008'] 2025-07-31 04:24:52,131 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-trans_left_vn-34185992 passed 2025-07-31 04:24:52,131 - DEBUG - ====Verifying policy data for ctest-trans_left_vn-34185992 in API_Server ====== 2025-07-31 04:24:52,132 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:24:52,144 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:24:52,164 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:24:52,299 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/140db3b8-7231-4a9c-ae16-b835272ad34c 2025-07-31 04:24:52,314 - DEBUG - =>VN ctest-trans_left_vn-34185992 has no policy to be verified 2025-07-31 04:24:52,314 - DEBUG - Verifying the vn in opserver 2025-07-31 04:24:52,314 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-trans_left_vn-34185992 virtual network link through opserver 10.0.0.41 2025-07-31 04:24:52,314 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-07-31 04:24:52,334 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-trans_left_vn-34185992', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-trans_left_vn-34185992?flat'} 2025-07-31 04:24:52,334 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-trans_left_vn-34185992 virtual network link through opserver 10.0.0.38 2025-07-31 04:24:52,334 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-07-31 04:24:52,354 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-trans_left_vn-34185992', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-trans_left_vn-34185992?flat'} 2025-07-31 04:24:52,354 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-trans_left_vn-34185992 virtual network link through opserver 10.0.0.61 2025-07-31 04:24:52,354 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-07-31 04:24:52,373 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-trans_left_vn-34185992', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-trans_left_vn-34185992?flat'} 2025-07-31 04:24:52,373 - INFO - Validated that VN default-domain:ctest-TestSvcHC-45474793:ctest-trans_left_vn-34185992 is found in opserver 2025-07-31 04:24:52,409 - DEBUG - Do not have enough data to verify VN in agent 2025-07-31 04:24:52,427 - DEBUG - VRF ids for VN ctest-trans_left_vn-34185992: {} 2025-07-31 04:24:52,824 - DEBUG - Response for create_network : {'network': {'id': '0fee58d6-028b-443c-8f5a-53fcd26b6470', 'name': 'ctest-trans_right_vn-12127630', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'project_id': '5d8e29c3f9ed4135910480d12bc16305', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-TestSvcHC-45474793', 'ctest-trans_right_vn-12127630'], 'port_security_enabled': True, 'description': ''}} 2025-07-31 04:24:53,086 - DEBUG - Response for create_subnet : {'subnet': {'id': '53414c4d-15e5-4595-8d80-aa692541e91e', 'name': '', 'tenant_id': '5d8e29c3f9ed4135910480d12bc16305', 'network_id': '0fee58d6-028b-443c-8f5a-53fcd26b6470', 'ip_version': 4, 'cidr': '124.177.85.192/26', 'allocation_pools': [{'start': '124.177.85.194', 'end': '124.177.85.254'}], 'gateway_ip': '124.177.85.193', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '124.177.85.194', 'tags': [], 'project_id': '5d8e29c3f9ed4135910480d12bc16305'}} 2025-07-31 04:24:53,114 - INFO - Created VN ctest-trans_right_vn-12127630 2025-07-31 04:24:53,129 - DEBUG - VN ctest-trans_right_vn-12127630 UUID is 0fee58d6-028b-443c-8f5a-53fcd26b6470 2025-07-31 04:24:53,227 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:24:53,242 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:24:53,271 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:24:53,398 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/0fee58d6-028b-443c-8f5a-53fcd26b6470 2025-07-31 04:24:53,423 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/0fee58d6-028b-443c-8f5a-53fcd26b6470 2025-07-31 04:24:53,443 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/13d5198d-80a1-40f5-a4c3-c5e5121c9fd7 2025-07-31 04:24:53,461 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/13d5198d-80a1-40f5-a4c3-c5e5121c9fd7 2025-07-31 04:24:53,476 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/965a1527-a3a3-4fed-8851-c199772ba133 2025-07-31 04:24:53,496 - DEBUG - Route Targets: ['target:64512:8000009'] 2025-07-31 04:24:53,496 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/0fee58d6-028b-443c-8f5a-53fcd26b6470 2025-07-31 04:24:53,523 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/13d5198d-80a1-40f5-a4c3-c5e5121c9fd7 2025-07-31 04:24:53,551 - INFO - Verified VN network id 15 for VN 0fee58d6-028b-443c-8f5a-53fcd26b6470 2025-07-31 04:24:53,552 - INFO - Verifications in API Server for VN ctest-trans_right_vn-12127630 passed 2025-07-31 04:24:53,552 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/0fee58d6-028b-443c-8f5a-53fcd26b6470 2025-07-31 04:24:53,578 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/13d5198d-80a1-40f5-a4c3-c5e5121c9fd7 2025-07-31 04:24:53,602 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/13d5198d-80a1-40f5-a4c3-c5e5121c9fd7 2025-07-31 04:24:53,629 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/965a1527-a3a3-4fed-8851-c199772ba133 2025-07-31 04:24:53,672 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-trans_right_vn-12127630', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1147952631214982204', 'uuid-lslong': '10329661040654771312'}, 'enable': 'true', 'created': '2025-07-31T04:24:52', 'last-modified': '2025-07-31T04:24:52', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.652531'} 2025-07-31 04:24:53,690 - DEBUG - Route Targets: ['target:64512:8000009'] 2025-07-31 04:24:53,701 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-trans_right_vn-12127630', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1147952631214982204', 'uuid-lslong': '10329661040654771312'}, 'enable': 'true', 'created': '2025-07-31T04:24:52', 'last-modified': '2025-07-31T04:24:52', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.684161'} 2025-07-31 04:24:53,713 - DEBUG - Route Targets: ['target:64512:8000009'] 2025-07-31 04:24:53,723 - DEBUG - Control-node 10.0.0.61 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-TestSvcHC-45474793:ctest-trans_right_vn-12127630', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-TestSvcHC-45474793', 'owner-access': '7', 'group': 'admin', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '1147952631214982204', 'uuid-lslong': '10329661040654771312'}, 'enable': 'true', 'created': '2025-07-31T04:24:52', 'last-modified': '2025-07-31T04:24:52', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.701286'} 2025-07-31 04:24:53,735 - DEBUG - Route Targets: ['target:64512:8000009'] 2025-07-31 04:24:53,735 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-trans_right_vn-12127630 passed 2025-07-31 04:24:53,736 - DEBUG - ====Verifying policy data for ctest-trans_right_vn-12127630 in API_Server ====== 2025-07-31 04:24:53,736 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:24:53,754 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:24:53,786 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:24:53,920 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/0fee58d6-028b-443c-8f5a-53fcd26b6470 2025-07-31 04:24:53,937 - DEBUG - =>VN ctest-trans_right_vn-12127630 has no policy to be verified 2025-07-31 04:24:53,937 - DEBUG - Verifying the vn in opserver 2025-07-31 04:24:53,937 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-trans_right_vn-12127630 virtual network link through opserver 10.0.0.41 2025-07-31 04:24:53,937 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2025-07-31 04:24:53,955 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-trans_right_vn-12127630', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-trans_right_vn-12127630?flat'} 2025-07-31 04:24:53,955 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-trans_right_vn-12127630 virtual network link through opserver 10.0.0.38 2025-07-31 04:24:53,955 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-07-31 04:24:53,975 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-trans_right_vn-12127630', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-trans_right_vn-12127630?flat'} 2025-07-31 04:24:53,975 - DEBUG - Verifying the default-domain:ctest-TestSvcHC-45474793:ctest-trans_right_vn-12127630 virtual network link through opserver 10.0.0.61 2025-07-31 04:24:53,975 - DEBUG - Requesting: http://10.0.0.61:8081/analytics/uves/virtual-networks 2025-07-31 04:24:53,995 - DEBUG - vn link and name as {'name': 'default-domain:ctest-TestSvcHC-45474793:ctest-trans_right_vn-12127630', 'href': 'http://10.0.0.61:8081/analytics/uves/virtual-network/default-domain:ctest-TestSvcHC-45474793:ctest-trans_right_vn-12127630?flat'} 2025-07-31 04:24:53,995 - INFO - Validated that VN default-domain:ctest-TestSvcHC-45474793:ctest-trans_right_vn-12127630 is found in opserver 2025-07-31 04:24:54,034 - DEBUG - Do not have enough data to verify VN in agent 2025-07-31 04:24:54,053 - DEBUG - VRF ids for VN ctest-trans_right_vn-12127630: {} 2025-07-31 04:24:55,502 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-2), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-2) 2025-07-31 04:24:57,127 - INFO - VM ([]) created on node: (an-jenkins-deploy-platform-ansible-os-3732-1), Zone: (nova:an-jenkins-deploy-platform-ansible-os-3732-1) 2025-07-31 04:24:57,127 - DEBUG - Creating service template: ['default-domain', 'ctest-service_template_1-58930728'] 2025-07-31 04:24:57,458 - DEBUG - Service template: ['default-domain', 'ctest-service_template_1-58930728'] created succesfully 2025-07-31 04:26:38,490 - DEBUG - Installing image tiny_trans_fw 2025-07-31 04:26:38,490 - DEBUG - Image info {'name': 'tinycore-net-transparent-fw.qcow2.gz', 'webserver': '', 'location': '/images/tinycore/', 'username': 'tc', 'password': 'secret', 'type': 'os', 'flavor': 'contrail_flavor_tiny', 'params': '--container-format bare --disk-format vmdk --property vmware_disktype="sparse" --property vmware_adaptertype="ide" --property hypervisor_type=qemu', 'name_docker': 'phusion-baseimage-enablesshd'} 2025-07-31 04:26:38,492 - DEBUG - Image build path file:///contrail-test/images/tinycore-net-transparent-fw.qcow2.gz 2025-07-31 04:26:38,492 - DEBUG - Download image from local file file:///contrail-test/images/tinycore-net-transparent-fw.qcow2.gz 2025-07-31 04:26:38,492 - DEBUG - Unzip image 2025-07-31 04:26:38,493 - DEBUG - Image download (local=True) (proxy=None): Execute cmd: gunzip -f /contrail-test/images/tinycore-net-transparent-fw.qcow2.gz 2025-07-31 04:26:38,537 - DEBUG - Try to create image in user context 2025-07-31 04:26:38,697 - DEBUG - Deleting service template: ['default-domain', 'ctest-service_template_1-58930728'] 2025-07-31 04:26:38,813 - INFO - Service template: ['default-domain', 'ctest-service_template_1-58930728'] deleted successfully. 2025-07-31 04:26:38,813 - INFO - Skip interface_detach for VM ctest-right_vm-52180568 in state BUILD 2025-07-31 04:26:38,813 - INFO - Deleting VM ctest-right_vm-52180568 2025-07-31 04:26:38,813 - ERROR - VM ctest-right_vm-52180568 has failed to come up 2025-07-31 04:26:38,814 - ERROR - Nova failed to get host of the VM 2025-07-31 04:26:38,814 - ERROR - VM ctest-right_vm-52180568 has failed to come up 2025-07-31 04:26:38,814 - ERROR - Nova failed to get host of the VM 2025-07-31 04:26:38,976 - INFO - Skip interface_detach for VM ctest-left_vm-73269375 in state BUILD 2025-07-31 04:26:38,976 - INFO - Deleting VM ctest-left_vm-73269375 2025-07-31 04:26:38,977 - ERROR - VM ctest-left_vm-73269375 has failed to come up 2025-07-31 04:26:38,977 - ERROR - Nova failed to get host of the VM 2025-07-31 04:26:38,977 - ERROR - VM ctest-left_vm-73269375 has failed to come up 2025-07-31 04:26:38,977 - ERROR - Nova failed to get host of the VM 2025-07-31 04:26:39,150 - INFO - Deleting VN ctest-trans_right_vn-12127630 2025-07-31 04:26:39,436 - DEBUG - Response for deleting network () 2025-07-31 04:26:39,436 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/13d5198d-80a1-40f5-a4c3-c5e5121c9fd7 2025-07-31 04:26:39,445 - DEBUG - Response Code: 404 2025-07-31 04:26:39,445 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:26:39,455 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:26:39,524 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:26:39,642 - INFO - Validated that VN ctest-trans_right_vn-12127630 is not found in API Server 2025-07-31 04:26:39,678 - DEBUG - VN ctest-trans_right_vn-12127630 is not present in Agent 10.0.0.27 2025-07-31 04:26:39,695 - DEBUG - VN ctest-trans_right_vn-12127630 is not present in Agent 10.0.0.57 2025-07-31 04:26:39,695 - INFO - Validated that VN ctest-trans_right_vn-12127630 is not in any agent 2025-07-31 04:26:39,751 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-trans_right_vn-12127630 info 2025-07-31 04:26:39,752 - INFO - Deleting VN ctest-trans_left_vn-34185992 2025-07-31 04:26:40,035 - DEBUG - Response for deleting network () 2025-07-31 04:26:40,035 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/87c33d3b-934a-43a1-9fe8-45ceae2c8310 2025-07-31 04:26:40,042 - DEBUG - Response Code: 404 2025-07-31 04:26:40,042 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:26:40,050 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:26:40,064 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:26:40,164 - INFO - Validated that VN ctest-trans_left_vn-34185992 is not found in API Server 2025-07-31 04:26:40,182 - DEBUG - VN ctest-trans_left_vn-34185992 is not present in Agent 10.0.0.27 2025-07-31 04:26:40,198 - DEBUG - VN ctest-trans_left_vn-34185992 is not present in Agent 10.0.0.57 2025-07-31 04:26:40,198 - INFO - Validated that VN ctest-trans_left_vn-34185992 is not in any agent 2025-07-31 04:26:40,254 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-trans_left_vn-34185992 info 2025-07-31 04:26:40,254 - INFO - Deleting VN ctest-right_vn-71731210 2025-07-31 04:26:40,444 - DEBUG - Response for deleting network () 2025-07-31 04:26:40,445 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/e6a1f278-a0dc-490b-9ee0-44190d508ed3 2025-07-31 04:26:40,454 - DEBUG - Response Code: 404 2025-07-31 04:26:40,454 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:26:40,460 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:26:40,474 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:26:40,583 - INFO - Validated that VN ctest-right_vn-71731210 is not found in API Server 2025-07-31 04:26:40,599 - DEBUG - VN ctest-right_vn-71731210 is not present in Agent 10.0.0.27 2025-07-31 04:26:40,615 - DEBUG - VN ctest-right_vn-71731210 is not present in Agent 10.0.0.57 2025-07-31 04:26:40,615 - INFO - Validated that VN ctest-right_vn-71731210 is not in any agent 2025-07-31 04:26:40,664 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-right_vn-71731210 info 2025-07-31 04:26:40,664 - INFO - Deleting VN ctest-left_vn-82355333 2025-07-31 04:26:40,848 - DEBUG - Response for deleting network () 2025-07-31 04:26:40,848 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/aa2c0b6c-49f3-4fa2-a9b4-f881ff5c5a5b 2025-07-31 04:26:40,855 - DEBUG - Response Code: 404 2025-07-31 04:26:40,855 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:26:40,861 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:26:40,926 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:26:41,036 - INFO - Validated that VN ctest-left_vn-82355333 is not found in API Server 2025-07-31 04:26:41,049 - DEBUG - VN ctest-left_vn-82355333 is not present in Agent 10.0.0.27 2025-07-31 04:26:41,066 - DEBUG - VN ctest-left_vn-82355333 is not present in Agent 10.0.0.57 2025-07-31 04:26:41,066 - INFO - Validated that VN ctest-left_vn-82355333 is not in any agent 2025-07-31 04:26:41,119 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-left_vn-82355333 info 2025-07-31 04:26:41,119 - INFO - Deleting VN ctest-mgmt_vn-81093085 2025-07-31 04:26:41,319 - DEBUG - Response for deleting network () 2025-07-31 04:26:41,320 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/371c6b36-9bdd-4a69-91de-c6c7fc415960 2025-07-31 04:26:41,326 - DEBUG - Response Code: 404 2025-07-31 04:26:41,326 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2025-07-31 04:26:41,332 - DEBUG - Requesting: http://10.0.0.41:8082/domain/34dad5b7-e30d-42c7-8ff4-7b0d7fa576d7 2025-07-31 04:26:41,359 - DEBUG - Requesting: http://10.0.0.41:8082/project/5d8e29c3-f9ed-4135-9104-80d12bc16305 2025-07-31 04:26:41,457 - INFO - Validated that VN ctest-mgmt_vn-81093085 is not found in API Server 2025-07-31 04:26:41,472 - DEBUG - VN ctest-mgmt_vn-81093085 is not present in Agent 10.0.0.27 2025-07-31 04:26:41,486 - DEBUG - VN ctest-mgmt_vn-81093085 is not present in Agent 10.0.0.57 2025-07-31 04:26:41,486 - INFO - Validated that VN ctest-mgmt_vn-81093085 is not in any agent 2025-07-31 04:26:41,534 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-mgmt_vn-81093085 info 2025-07-31 04:26:42,893 - ERROR - 
Traceback (most recent call last):
  File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
    result = function(self, *args, **kwargs)
  File "/contrail-test/scripts/svc_health_check/test_svc_health_check.py", line 137, in test_svc_trans_segment_right_hc_fail_svm
    assert self.svc_hc_fail_svm_common()
  File "/contrail-test/scripts/svc_health_check/test_svc_health_check.py", line 180, in svc_hc_fail_svm_common
    create_svms=True, max_inst=1)
  File "/contrail-test/common/servicechain/verify.py", line 320, in verify_svc_chain
    ret_dict = svc_chain_info or self.config_svc_chain(*args, **kwargs)
  File "/contrail-test/common/servicechain/config.py", line 533, in config_svc_chain
    max_inst=max_inst)
  File "/contrail-test/common/servicechain/config.py", line 335, in create_service_vms
    zone=random.choice(valid_zones))
  File "/contrail-test/common/servicechain/config.py", line 161, in config_vm_only
    flavor=flavor, zone=zone, node_name=node_name,**kwargs)
  File "/contrail-test/common/servicechain/config.py", line 203, in config_vm
    flavor=flavor, vn_objs=vns, count=count, zone=zone,**kwargs))
  File "/usr/local/lib/python3.6/site-packages/testtools/testcase.py", line 756, in useFixture
    reraise(*exc_info)
  File "/usr/local/lib/python3.6/site-packages/testtools/_compat3x.py", line 16, in reraise
    raise exc_obj.with_traceback(exc_tb)
  File "/usr/local/lib/python3.6/site-packages/testtools/testcase.py", line 731, in useFixture
    fixture.setUp()
  File "/contrail-test/fixtures/vm_test.py", line 188, in setUp
    self.create()
  File "/contrail-test/fixtures/vm_test.py", line 240, in create
    fixed_ips=self.fixed_ips)
  File "/contrail-test/fixtures/openstack.py", line 118, in create_vm
    return self.nova_h.create_vm(vm_name=vm_name, image_name=image_name, vn_ids=vn_ids, **kwargs)
  File "/contrail-test/fixtures/nova_test.py", line 577, in create_vm
    image = self.get_image(image_name=image_name)
  File "/contrail-test/fixtures/nova_test.py", line 198, in get_image
    self._install_image(image_name=image_name)
  File "/contrail-test/fixtures/nova_test.py", line 338, in _install_image
    return self.copy_and_glance(build_path, image_name, params)
  File "/contrail-test/fixtures/nova_test.py", line 407, in copy_and_glance
    image_id = self.glance_h.create_image(generic_image_name, image_path_real, **params)
  File "/contrail-test/fixtures/glance_test.py", line 50, in create_image
    self.upload_image(obj['id'], filename)
  File "/contrail-test/fixtures/glance_test.py", line 54, in upload_image
    self.obj.images.upload(uuid, open(filename, 'rb'))
FileNotFoundError: [Errno 2] No such file or directory: '/contrail-test/images/tinycore-net-transparent-fw.qcow2'
2025-07-31 04:26:42,922 - DEBUG - No XMPP flaps were noticed during the test 2025-07-31 04:26:42,922 - INFO - 2025-07-31 04:26:42,922 - INFO - END TEST : test_svc_trans_segment_right_hc_fail_svm : FAILED[0:01:57] 2025-07-31 04:26:42,923 - INFO - -------------------------------------------------------------------------------- 2025-07-31 04:26:43,893 - INFO - Deleted project: ctest-TestSvcHC-45474793, ID : 5d8e29c3-f9ed-4135-9104-80d12bc16305