2025-07-23 17:01:31,267 - INFO - Domain Default found not creating 2025-07-23 17:01:31,410 - INFO - Project ctest-FatFlowAggr-43339952 not found, creating it 2025-07-23 17:01:32,020 - INFO - Created Project:ctest-FatFlowAggr-43339952, ID : 77348154-4aa4-4cb2-a018-c31ad6bfe6b2 2025-07-23 17:01:33,494 - INFO - ================================================================================ 2025-07-23 17:01:33,494 - INFO - STARTING TEST : test_fat_flow_aggr_dest_icmp_intra_vn_inter_node 2025-07-23 17:01:33,494 - INFO - TEST DESCRIPTION : : Description: Verify fat flow prefix aggr dest (IPv4) for intra-vn inter-node Steps: 1. Create 1 VN and launch 3 VMs.2 client VMs on same node and server VM on different node. Client 1 in subnet 1, Client 2 in the next subnet. 2. On server VM, config fat flow aggr prefix dest len 25 for ICMP port 0. 3. From both the client VMs, send ICMP traffic to the server VM twice with diff. src ports Pass criteria: 1. On the remote CN, expect 2 pairs ( 1 for client 1, 1 for client 2) of fat flows with prefix aggregated for the src IPs (VM to fabric, Prefix Aggr Dest: Aggregation happens for SRC IPs) 2. On client VM compute nodes, expect 4 pairs of flows and on server compute, expect 2 pairs of flows 3. On server compute node, flow's source port should be 0 for fat flows Maintainer: Ankitja@juniper.net 2025-07-23 17:01:34,817 - DEBUG - Nothing to compare xmpp stats {'10.0.0.50': {'10.20.0.254': '0', '10.20.0.17': '0'}, '10.0.0.130': {'10.20.0.17': '0', '10.20.0.254': '0'}} with 2025-07-23 17:01:34,817 - INFO - Initial checks done. Running the testcase now 2025-07-23 17:01:34,817 - INFO - 2025-07-23 17:01:34,852 - DEBUG - Services list from nova: [, , , ] 2025-07-23 17:01:34,852 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3704-2', 'an-jenkins-deploy-platform-ansible-os-3704-1']} 2025-07-23 17:01:35,616 - DEBUG - Response for create_network : {'network': {'id': 'e9eeba0c-fbee-4dc2-b19f-574003e524ac', 'name': 'ctest-vn-10006515', 'tenant_id': '773481544aa44cb2a018c31ad6bfe6b2', 'project_id': '773481544aa44cb2a018c31ad6bfe6b2', 'admin_state_up': True, 'shared': False, 'status': 'ACTIVE', 'router:external': False, 'mtu': 0, 'subnets': [], 'fq_name': ['default-domain', 'ctest-FatFlowAggr-43339952', 'ctest-vn-10006515'], 'port_security_enabled': True, 'description': ''}} 2025-07-23 17:01:35,877 - DEBUG - Response for create_subnet : {'subnet': {'id': '8713bdab-7553-4836-a709-2941f1630caf', 'name': '', 'tenant_id': '773481544aa44cb2a018c31ad6bfe6b2', 'network_id': 'e9eeba0c-fbee-4dc2-b19f-574003e524ac', 'ip_version': 4, 'cidr': '204.88.186.64/26', 'allocation_pools': [{'start': '204.88.186.66', 'end': '204.88.186.126'}], 'gateway_ip': '204.88.186.65', 'enable_dhcp': True, 'ipv6_ra_mode': None, 'ipv6_address_mode': None, 'dns_nameservers': [], 'host_routes': [], 'dns_server_address': '204.88.186.66', 'tags': [], 'project_id': '773481544aa44cb2a018c31ad6bfe6b2'}} 2025-07-23 17:01:35,901 - INFO - Created VN ctest-vn-10006515 2025-07-23 17:01:35,919 - DEBUG - VN ctest-vn-10006515 UUID is e9eeba0c-fbee-4dc2-b19f-574003e524ac 2025-07-23 17:01:36,219 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-07-23 17:01:36,524 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-07-23 17:01:36,557 - DEBUG - Requesting: http://10.0.0.23:8082/domain/85afe8be-97e7-4e27-9667-ee7272dde692 2025-07-23 17:01:36,581 - DEBUG - Requesting: http://10.0.0.23:8082/project/77348154-4aa4-4cb2-a018-c31ad6bfe6b2 2025-07-23 17:01:36,684 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/e9eeba0c-fbee-4dc2-b19f-574003e524ac 2025-07-23 17:01:36,697 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/e9eeba0c-fbee-4dc2-b19f-574003e524ac 2025-07-23 17:01:36,707 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/fd065155-4c46-465e-b5ff-443d84a91038 2025-07-23 17:01:36,719 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/fd065155-4c46-465e-b5ff-443d84a91038 2025-07-23 17:01:36,728 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/a6cddb13-d0d3-4a95-b45d-1d717f732353 2025-07-23 17:01:36,737 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/c7c158ae-1302-43be-95be-37fd24b62c7f 2025-07-23 17:01:36,746 - DEBUG - Route Targets: ['target:64512:10000', 'target:64512:8000005'] 2025-07-23 17:01:36,746 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/e9eeba0c-fbee-4dc2-b19f-574003e524ac 2025-07-23 17:01:36,756 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/fd065155-4c46-465e-b5ff-443d84a91038 2025-07-23 17:01:36,776 - INFO - Verified VN network id 12 for VN e9eeba0c-fbee-4dc2-b19f-574003e524ac 2025-07-23 17:01:36,776 - INFO - Verifications in API Server for VN ctest-vn-10006515 passed 2025-07-23 17:01:36,777 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/e9eeba0c-fbee-4dc2-b19f-574003e524ac 2025-07-23 17:01:36,788 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/fd065155-4c46-465e-b5ff-443d84a91038 2025-07-23 17:01:36,798 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/fd065155-4c46-465e-b5ff-443d84a91038 2025-07-23 17:01:36,806 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/a6cddb13-d0d3-4a95-b45d-1d717f732353 2025-07-23 17:01:36,813 - DEBUG - Requesting: http://10.0.0.23:8082/route-target/c7c158ae-1302-43be-95be-37fd24b62c7f 2025-07-23 17:01:36,827 - DEBUG - Control-node 10.0.0.23 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-FatFlowAggr-43339952:ctest-vn-10006515', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-FatFlowAggr-43339952', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16856615020223417794', 'uuid-lslong': '12799044598465111212'}, 'enable': 'true', 'created': '2025-07-23T17:01:35', 'last-modified': '2025-07-23T17:01:36', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.769425'} 2025-07-23 17:01:36,833 - DEBUG - Route Targets: ['target:64512:10000', 'target:64512:8000005'] 2025-07-23 17:01:36,841 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-FatFlowAggr-43339952:ctest-vn-10006515', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-FatFlowAggr-43339952', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16856615020223417794', 'uuid-lslong': '12799044598465111212'}, 'enable': 'true', 'created': '2025-07-23T17:01:35', 'last-modified': '2025-07-23T17:01:36', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.788963'} 2025-07-23 17:01:36,848 - DEBUG - Route Targets: ['target:64512:10000', 'target:64512:8000005'] 2025-07-23 17:01:36,855 - DEBUG - Control-node 10.0.0.73 : VN object is : {'node_name': 'virtual-network:default-domain:ctest-FatFlowAggr-43339952:ctest-vn-10006515', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'ctest-FatFlowAggr-43339952', 'owner-access': '7', 'group': 'reader', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16856615020223417794', 'uuid-lslong': '12799044598465111212'}, 'enable': 'true', 'created': '2025-07-23T17:01:35', 'last-modified': '2025-07-23T17:01:36', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.804490'} 2025-07-23 17:01:36,862 - DEBUG - Route Targets: ['target:64512:10000', 'target:64512:8000005'] 2025-07-23 17:01:36,862 - INFO - On all control nodes, Config, RI and RT verification for VN ctest-vn-10006515 passed 2025-07-23 17:01:36,862 - DEBUG - ====Verifying policy data for ctest-vn-10006515 in API_Server ====== 2025-07-23 17:01:36,862 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-07-23 17:01:36,869 - DEBUG - Requesting: http://10.0.0.23:8082/domain/85afe8be-97e7-4e27-9667-ee7272dde692 2025-07-23 17:01:36,881 - DEBUG - Requesting: http://10.0.0.23:8082/project/77348154-4aa4-4cb2-a018-c31ad6bfe6b2 2025-07-23 17:01:36,974 - DEBUG - Requesting: http://10.0.0.23:8082/virtual-network/e9eeba0c-fbee-4dc2-b19f-574003e524ac 2025-07-23 17:01:36,984 - DEBUG - =>VN ctest-vn-10006515 has no policy to be verified 2025-07-23 17:01:36,984 - DEBUG - Verifying the vn in opserver 2025-07-23 17:01:36,984 - DEBUG - Verifying the default-domain:ctest-FatFlowAggr-43339952:ctest-vn-10006515 virtual network link through opserver 10.0.0.23 2025-07-23 17:01:36,985 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-07-23 17:01:37,282 - DEBUG - Requesting: http://10.0.0.23:8081/analytics/uves/virtual-networks 2025-07-23 17:01:37,340 - DEBUG - vn link and name as {'name': 'default-domain:ctest-FatFlowAggr-43339952:ctest-vn-10006515', 'href': 'http://10.0.0.23:8081/analytics/uves/virtual-network/default-domain:ctest-FatFlowAggr-43339952:ctest-vn-10006515?flat'} 2025-07-23 17:01:37,340 - DEBUG - Verifying the default-domain:ctest-FatFlowAggr-43339952:ctest-vn-10006515 virtual network link through opserver 10.0.0.249 2025-07-23 17:01:37,340 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2025-07-23 17:01:37,634 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2025-07-23 17:01:37,702 - DEBUG - vn link and name as {'name': 'default-domain:ctest-FatFlowAggr-43339952:ctest-vn-10006515', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:ctest-FatFlowAggr-43339952:ctest-vn-10006515?flat'} 2025-07-23 17:01:37,702 - DEBUG - Verifying the default-domain:ctest-FatFlowAggr-43339952:ctest-vn-10006515 virtual network link through opserver 10.0.0.73 2025-07-23 17:01:37,702 - DEBUG - Requesting: http://10.0.0.73:8081/analytics/uves/virtual-networks 2025-07-23 17:01:38,000 - DEBUG - Requesting: http://10.0.0.73:8081/analytics/uves/virtual-networks 2025-07-23 17:01:38,058 - DEBUG - vn link and name as {'name': 'default-domain:ctest-FatFlowAggr-43339952:ctest-vn-10006515', 'href': 'http://10.0.0.73:8081/analytics/uves/virtual-network/default-domain:ctest-FatFlowAggr-43339952:ctest-vn-10006515?flat'} 2025-07-23 17:01:38,058 - INFO - Validated that VN default-domain:ctest-FatFlowAggr-43339952:ctest-vn-10006515 is found in opserver 2025-07-23 17:01:38,100 - DEBUG - Do not have enough data to verify VN in agent 2025-07-23 17:01:38,119 - DEBUG - VRF ids for VN ctest-vn-10006515: {} 2025-07-23 17:01:38,165 - DEBUG - Hosts: {'nova': ['an-jenkins-deploy-platform-ansible-os-3704-2', 'an-jenkins-deploy-platform-ansible-os-3704-1']} 2025-07-23 17:03:18,533 - DEBUG - Installing image ubuntu 2025-07-23 17:03:18,533 - DEBUG - Image info {'name': 'ubuntu.vmdk.gz', 'webserver': '', 'location': '/images/converts/', 'username': 'ubuntu', 'password': 'ubuntu', 'type': 'os', 'flavor': 'contrail_flavor_tiny', 'params': '--container-format bare --disk-format vmdk --property vmware_disktype="sparse" --property vmware_adaptertype="ide" --property hw_scsi_model=virtio-scsi --property hw_disk_bus_model=virtio-scsi --property hw_disk_bus=virtio', 'name_docker': 'phusion-baseimage-enablesshd'} 2025-07-23 17:03:18,533 - DEBUG - Image build path file:///contrail-test/images/ubuntu.vmdk.gz 2025-07-23 17:03:18,533 - DEBUG - Download image from local file file:///contrail-test/images/ubuntu.vmdk.gz 2025-07-23 17:03:18,533 - DEBUG - Unzip image 2025-07-23 17:03:18,533 - DEBUG - Image download (local=True) (proxy=None): Execute cmd: gunzip -f /contrail-test/images/ubuntu.vmdk.gz 2025-07-23 17:03:18,554 - DEBUG - Try to create image in user context 2025-07-23 17:03:18,626 - DEBUG - Deleting RT for VN ctest-vn-10006515 2025-07-23 17:03:18,759 - INFO - Deleting VN ctest-vn-10006515 2025-07-23 17:03:18,910 - DEBUG - Response for deleting network () 2025-07-23 17:03:18,910 - DEBUG - Requesting: http://10.0.0.23:8082/routing-instance/fd065155-4c46-465e-b5ff-443d84a91038 2025-07-23 17:03:18,916 - DEBUG - Response Code: 404 2025-07-23 17:03:18,916 - DEBUG - Requesting: http://10.0.0.23:8082/domains 2025-07-23 17:03:18,922 - DEBUG - Requesting: http://10.0.0.23:8082/domain/85afe8be-97e7-4e27-9667-ee7272dde692 2025-07-23 17:03:18,935 - DEBUG - Requesting: http://10.0.0.23:8082/project/77348154-4aa4-4cb2-a018-c31ad6bfe6b2 2025-07-23 17:03:19,036 - INFO - Validated that VN ctest-vn-10006515 is not found in API Server 2025-07-23 17:03:19,062 - DEBUG - VN ctest-vn-10006515 is not present in Agent 10.0.0.50 2025-07-23 17:03:19,075 - DEBUG - VN ctest-vn-10006515 is not present in Agent 10.0.0.130 2025-07-23 17:03:19,075 - INFO - Validated that VN ctest-vn-10006515 is not in any agent 2025-07-23 17:03:19,117 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn-10006515 info 2025-07-23 17:03:20,405 - ERROR - 
Traceback (most recent call last):
  File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
    result = function(self, *args, **kwargs)
  File "/contrail-test/scripts/vrouter/test_fat_flow_aggr.py", line 61, in test_fat_flow_aggr_dest_icmp_intra_vn_inter_node
    dual=ipv6, prefix_length6=prefix_length6, only_v6=only_v6)
  File "/contrail-test/common/vrouter/base.py", line 383, in fat_flow_with_prefix_aggr
    node_name=compute_hosts[0])
  File "/contrail-test/common/vrouter/base.py", line 179, in create_vms
    *args, **kwargs
  File "/contrail-test/common/base.py", line 281, in create_vm
    **kwargs)
  File "/contrail-test/common/base.py", line 254, in create_only_vm
    vm_obj.setUp()
  File "/contrail-test/fixtures/vm_test.py", line 188, in setUp
    self.create()
  File "/contrail-test/fixtures/vm_test.py", line 240, in create
    fixed_ips=self.fixed_ips)
  File "/contrail-test/fixtures/openstack.py", line 118, in create_vm
    return self.nova_h.create_vm(vm_name=vm_name, image_name=image_name, vn_ids=vn_ids, **kwargs)
  File "/contrail-test/fixtures/nova_test.py", line 580, in create_vm
    image = self.get_image(image_name=image_name)
  File "/contrail-test/fixtures/nova_test.py", line 199, in get_image
    self._install_image(image_name=image_name)
  File "/contrail-test/fixtures/nova_test.py", line 339, in _install_image
    return self.copy_and_glance(build_path, image_name, params)
  File "/contrail-test/fixtures/nova_test.py", line 410, in copy_and_glance
    image_id = self.glance_h.create_image(generic_image_name, image_path_real, **params)
  File "/contrail-test/fixtures/glance_test.py", line 50, in create_image
    self.upload_image(obj['id'], filename)
  File "/contrail-test/fixtures/glance_test.py", line 54, in upload_image
    self.obj.images.upload(uuid, open(filename, 'rb'))
FileNotFoundError: [Errno 2] No such file or directory: '/contrail-test/images/ubuntu.vmdk'
2025-07-23 17:03:20,422 - DEBUG - No XMPP flaps were noticed during the test 2025-07-23 17:03:20,422 - INFO - 2025-07-23 17:03:20,422 - INFO - END TEST : test_fat_flow_aggr_dest_icmp_intra_vn_inter_node : FAILED[0:01:47] 2025-07-23 17:03:20,422 - INFO - -------------------------------------------------------------------------------- 2025-07-23 17:03:21,134 - INFO - Deleted project: ctest-FatFlowAggr-43339952, ID : 77348154-4aa4-4cb2-a018-c31ad6bfe6b2