The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
result = function(self, *args, **kwargs)
File "/contrail-test/serial_scripts/vm_regression/test_vm_serial.py", line 435, in test_process_restart_in_policy_between_vns
assert result[0], result[1]
AssertionError: verify_process_and_connection_infos_config status is invalid for contrail-device-manager
{'NodeStatus': {'process_status': [{'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754916642241911, 'system_mem_usage': {'total': 16175640, 'used': 6908036, 'free': 256848, 'buffers': 287376, 'cached': 8723380, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9276888, 'partition_space_available_1k': 71813596, 'percentage_partition_space_used': 11}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 96416, 'cpu_share': 0.0, 'mem_res': 94520}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 102860, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 52172, 'cpu_share': 0.01, 'mem_res': 40192}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.3, 'five_min_avg': 1.33, 'fifteen_min_avg': 1.36, 'cpu_share': 6.5, 'node_type': 'config-node'}, 'deleted': False}, 'UVEAlarms': {'__T': 1754916470740432, 'alarms': [{'type': 'default-global-system-config:system-defined-conf-incorrect', 'alarm_rules': {'or_list': [{'and_list': [{'condition': {'operation': '==', 'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'variables': []}, 'match': [{'json_operand1_value': 'null', 'json_variables': {}}]}]}]}, 'ack': False, 'timestamp': 1754916470740255, 'severity': 1, 'token': 'eyJob3N0X2lwIjogIjEwLjAuMC4zOCIsICJodHRwX3BvcnQiOiA1OTk1LCAidGltZXN0YW1wIjogMTc1NDkxNjQ3MDc0MDI1NX0=', 'description': 'ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.'}]}}
Traceback (most recent call last):
File "/contrail-test/serial_scripts/vm_regression/test_vm_serial.py", line 470, in verification_after_process_restart_in_rr
self.analytics_obj.verify_process_and_connection_infos_config()
File "/contrail-test/tcutils/collector/analytics_tests.py", line 4128, in verify_process_and_connection_infos_config
for k in module_connection_dict:
AssertionError: status is invalid for contrail-device-manager
{'NodeStatus': {'process_status': [{'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754916642241911, 'system_mem_usage': {'total': 16175640, 'used': 6908036, 'free': 256848, 'buffers': 287376, 'cached': 8723380, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9276888, 'partition_space_available_1k': 71813596, 'percentage_partition_space_used': 11}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 96416, 'cpu_share': 0.0, 'mem_res': 94520}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 102860, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 52172, 'cpu_share': 0.01, 'mem_res': 40192}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.3, 'five_min_avg': 1.33, 'fifteen_min_avg': 1.36, 'cpu_share': 6.5, 'node_type': 'config-node'}, 'deleted': False}, 'UVEAlarms': {'__T': 1754916470740432, 'alarms': [{'type': 'default-global-system-config:system-defined-conf-incorrect', 'alarm_rules': {'or_list': [{'and_list': [{'condition': {'operation': '==', 'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'variables': []}, 'match': [{'json_operand1_value': 'null', 'json_variables': {}}]}]}]}, 'ack': False, 'timestamp': 1754916470740255, 'severity': 1, 'token': 'eyJob3N0X2lwIjogIjEwLjAuMC4zOCIsICJodHRwX3BvcnQiOiA1OTk1LCAidGltZXN0YW1wIjogMTc1NDkxNjQ3MDc0MDI1NX0=', 'description': 'ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.'}]}}
2025-08-11 12:51:17,817 - DEBUG - Skipping xmpp flap check
2025-08-11 12:51:17,817 - INFO -
2025-08-11 12:51:17,817 - INFO - END TEST : test_process_restart_in_policy_between_vns : FAILED[0:05:32]
2025-08-11 12:51:17,817 - INFO - --------------------------------------------------------------------------------
2025-08-11 12:51:17,820 - INFO - ================================================================================
2025-08-11 12:51:17,821 - INFO - STARTING TEST : test_underlay_broadcast_traffic_handling
2025-08-11 12:51:17,821 - INFO - TEST DESCRIPTION : Test the underlay brocast traffic handling by vrouter. (Bug-1545229).
1. Send broadcast traffic from one compute node.
2. Other compute in same subnet should receive that traffic.
3. Receiving compute should treat this traffic as underlay.
4. Compute should not replicate the packet and send the copy back.
Pass criteria: Step 3-4 should pass
Maintainer : chhandak@juniper.net
2025-08-11 12:51:19,110 - DEBUG - Skipping xmpp flap check
2025-08-11 12:51:19,110 - INFO - Initial checks done. Running the testcase now
2025-08-11 12:51:19,110 - INFO -
2025-08-11 12:51:19,500 - INFO - Agent interface name: ens6
2025-08-11 12:51:19,569 - DEBUG - Executing command: sudo tcpdump -nni ens6 -U host 10.20.0.255 -w /tmp/ens6_ctest-random-50587116.pcap
2025-08-11 12:51:24,922 - INFO - Pinging broacast address 10.20.0.255 from compute 10.20.0.193
2025-08-11 12:51:39,220 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}')
2025-08-11 12:51:41,358 - DEBUG - Executing command: sudo tcpdump -nnr /tmp/ens6_ctest-random-50587116.pcap | grep -c length
2025-08-11 12:51:41,369 - DEBUG - STDOUT: 10
2025-08-11 12:51:41,369 - DEBUG - STDERR: reading from file /tmp/ens6_ctest-random-50587116.pcap, link-type EN10MB (Ethernet), snapshot length 262144
2025-08-11 12:51:41,369 - INFO - 10 packets are found in tcpdump output as expected
2025-08-11 12:51:41,370 - DEBUG - Executing command: sudo kill $(ps -ef|grep tcpdump | grep pcap| awk '{print $2}')
2025-08-11 12:51:43,413 - INFO - Packet count matched: Compute 10.0.0.41 has receive only 10 packet from source IP 10.20.0.255. No duplicate packet seen
2025-08-11 12:51:44,644 - DEBUG - Skipping xmpp flap check
2025-08-11 12:51:44,644 - INFO - END TEST : test_underlay_broadcast_traffic_handling : PASSED[0:00:27]
2025-08-11 12:51:44,644 - INFO - --------------------------------------------------------------------------------
2025-08-11 12:51:45,329 - INFO - Deleted project: ctest-TestBasicVMVN0-68776849, ID : c0ce5cde-dd6e-472e-8d94-cd73e9f2261b