2025-09-08 02:20:32,736 - INFO - Using existing project ['default-domain', 'admin'](d3c9e274-58c1-485b-8908-c8c223fb21f3)
2025-09-08 02:20:32,754 - INFO - ================================================================================
2025-09-08 02:20:32,754 - INFO - STARTING TEST : test_contrail_status
2025-09-08 02:20:32,754 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active
2025-09-08 02:20:33,521 - DEBUG - Nothing to compare xmpp stats {'10.0.0.65': {'10.20.0.14': '0', '10.20.0.129': '0'}, '10.0.0.23': {'10.20.0.25': '0', '10.20.0.129': '0'}} with
2025-09-08 02:20:33,522 - INFO - Initial checks done. Running the testcase now
2025-09-08 02:20:33,522 - INFO -
2025-09-08 02:20:33,601 - INFO - 10.0.0.41
2025-09-08 02:20:33,601 - INFO - config
2025-09-08 02:20:33,606 - INFO - config-nodemgr:active
2025-09-08 02:20:33,611 - INFO - api-server:active
2025-09-08 02:20:33,614 - INFO - schema:backup
2025-09-08 02:20:33,617 - INFO - svc-monitor:backup
2025-09-08 02:20:33,619 - INFO - device-manager:backup
2025-09-08 02:20:33,619 - INFO - config-database
2025-09-08 02:20:33,619 - INFO - config-cassandra:active
2025-09-08 02:20:33,619 - INFO - config-zookeeper:active
2025-09-08 02:20:33,619 - INFO - config-rabbitmq:active
2025-09-08 02:20:33,619 - INFO - control
2025-09-08 02:20:33,625 - INFO - control-nodemgr:active
2025-09-08 02:20:33,632 - INFO - control:active
2025-09-08 02:20:33,632 - INFO - named:active
2025-09-08 02:20:33,639 - INFO - dns:active
2025-09-08 02:20:33,639 - INFO - analytics
2025-09-08 02:20:33,645 - INFO - analytics-nodemgr:active
2025-09-08 02:20:33,649 - INFO - analytics-api:active
2025-09-08 02:20:33,654 - INFO - collector:active
2025-09-08 02:20:33,654 - INFO - analytics-database
2025-09-08 02:20:33,654 - INFO - analytics-cassandra:active
2025-09-08 02:20:33,659 - INFO - analyticsdb-nodemgr:active
2025-09-08 02:20:33,664 - INFO - query-engine:active
2025-09-08 02:20:33,664 - INFO - webui
2025-09-08 02:20:33,664 - INFO - webui:active
2025-09-08 02:20:33,664 - INFO - webui-middleware:active
2025-09-08 02:20:33,664 - INFO - redis:active
2025-09-08 02:20:33,664 - INFO - analytics_snmp
2025-09-08 02:20:33,669 - INFO - snmp-collector:active
2025-09-08 02:20:33,669 - INFO - snmp-topology:active
2025-09-08 02:20:33,669 - INFO - analytics_alarm
2025-09-08 02:20:33,669 - INFO - alarmgen:active
2025-09-08 02:20:33,724 - INFO - 10.0.0.27
2025-09-08 02:20:33,724 - INFO - config
2025-09-08 02:20:33,730 - INFO - config-nodemgr:active
2025-09-08 02:20:33,734 - INFO - api-server:active
2025-09-08 02:20:33,738 - INFO - schema:active
2025-09-08 02:20:33,742 - INFO - svc-monitor:active
2025-09-08 02:20:33,746 - INFO - device-manager:active
2025-09-08 02:20:33,746 - INFO - config-database
2025-09-08 02:20:33,746 - INFO - config-cassandra:active
2025-09-08 02:20:33,747 - INFO - config-zookeeper:active
2025-09-08 02:20:33,747 - INFO - config-rabbitmq:active
2025-09-08 02:20:33,747 - INFO - control
2025-09-08 02:20:33,751 - INFO - control-nodemgr:active
2025-09-08 02:20:33,756 - INFO - control:active
2025-09-08 02:20:33,756 - INFO - named:active
2025-09-08 02:20:33,761 - INFO - dns:active
2025-09-08 02:20:33,761 - INFO - analytics
2025-09-08 02:20:33,765 - INFO - analytics-nodemgr:active
2025-09-08 02:20:33,769 - INFO - analytics-api:active
2025-09-08 02:20:33,774 - INFO - collector:active
2025-09-08 02:20:33,774 - INFO - analytics-database
2025-09-08 02:20:33,774 - INFO - analytics-cassandra:active
2025-09-08 02:20:33,779 - INFO - analyticsdb-nodemgr:active
2025-09-08 02:20:33,785 - INFO - query-engine:active
2025-09-08 02:20:33,785 - INFO - webui
2025-09-08 02:20:33,785 - INFO - webui:active
2025-09-08 02:20:33,785 - INFO - webui-middleware:active
2025-09-08 02:20:33,785 - INFO - redis:active
2025-09-08 02:20:33,785 - INFO - analytics_snmp
2025-09-08 02:20:33,789 - INFO - snmp-collector:active
2025-09-08 02:20:33,790 - INFO - snmp-topology:active
2025-09-08 02:20:33,790 - INFO - analytics_alarm
2025-09-08 02:20:33,790 - INFO - alarmgen:active
2025-09-08 02:20:33,846 - INFO - 10.0.0.38
2025-09-08 02:20:33,846 - INFO - config
2025-09-08 02:20:33,853 - INFO - config-nodemgr:active
2025-09-08 02:20:33,858 - INFO - api-server:active
2025-09-08 02:20:33,861 - INFO - schema:backup
2025-09-08 02:20:33,864 - INFO - svc-monitor:backup
2025-09-08 02:20:33,867 - INFO - device-manager:backup
2025-09-08 02:20:33,868 - INFO - config-database
2025-09-08 02:20:33,868 - INFO - config-cassandra:active
2025-09-08 02:20:33,868 - INFO - config-zookeeper:active
2025-09-08 02:20:33,868 - INFO - config-rabbitmq:active
2025-09-08 02:20:33,868 - INFO - control
2025-09-08 02:20:33,873 - INFO - control-nodemgr:active
2025-09-08 02:20:33,878 - INFO - control:active
2025-09-08 02:20:33,878 - INFO - named:active
2025-09-08 02:20:33,883 - INFO - dns:active
2025-09-08 02:20:33,883 - INFO - analytics
2025-09-08 02:20:33,889 - INFO - analytics-nodemgr:active
2025-09-08 02:20:33,894 - INFO - analytics-api:active
2025-09-08 02:20:33,899 - INFO - collector:active
2025-09-08 02:20:33,899 - INFO - analytics-database
2025-09-08 02:20:33,899 - INFO - analytics-cassandra:active
2025-09-08 02:20:33,905 - INFO - analyticsdb-nodemgr:active
2025-09-08 02:20:33,910 - INFO - query-engine:active
2025-09-08 02:20:33,910 - INFO - webui
2025-09-08 02:20:33,911 - INFO - webui:active
2025-09-08 02:20:33,911 - INFO - webui-middleware:active
2025-09-08 02:20:33,911 - INFO - redis:active
2025-09-08 02:20:33,911 - INFO - analytics_snmp
2025-09-08 02:20:33,915 - INFO - snmp-collector:active
2025-09-08 02:20:33,915 - INFO - snmp-topology:active
2025-09-08 02:20:33,915 - INFO - analytics_alarm
2025-09-08 02:20:33,915 - INFO - alarmgen:active
2025-09-08 02:20:34,003 - INFO - 10.0.0.65
2025-09-08 02:20:34,003 - INFO - vrouter
2025-09-08 02:20:34,013 - INFO - vrouter-nodemgr:active
2025-09-08 02:20:34,020 - INFO - agent:active
2025-09-08 02:20:34,139 - INFO - 10.0.0.23
2025-09-08 02:20:34,139 - INFO - vrouter
2025-09-08 02:20:34,148 - INFO - vrouter-nodemgr:active
2025-09-08 02:20:34,161 - INFO - agent:active
2025-09-08 02:20:34,161 - INFO - All the contrail services are up on all nodes
2025-09-08 02:20:34,161 - INFO - contrail-status passed
2025-09-08 02:20:34,909 - DEBUG - No XMPP flaps were noticed during the test
2025-09-08 02:20:34,909 - INFO - END TEST : test_contrail_status : PASSED[0:00:02]
2025-09-08 02:20:34,909 - INFO - --------------------------------------------------------------------------------
2025-09-08 04:16:02,944 - INFO - Using existing project ['default-domain', 'admin'](d3c9e274-58c1-485b-8908-c8c223fb21f3)
2025-09-08 04:16:02,956 - INFO - ================================================================================
2025-09-08 04:16:02,956 - INFO - STARTING TEST : test_contrail_status
2025-09-08 04:16:02,956 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active
2025-09-08 04:16:04,313 - DEBUG - Skipping xmpp flap check
2025-09-08 04:16:04,313 - INFO - Initial checks done. Running the testcase now
2025-09-08 04:16:04,313 - INFO -
2025-09-08 04:16:04,586 - INFO - 10.0.0.41
2025-09-08 04:16:04,586 - INFO - config
2025-09-08 04:16:04,594 - INFO - config-nodemgr:active
2025-09-08 04:16:04,600 - INFO - api-server:active
2025-09-08 04:16:04,603 - INFO - schema:backup
2025-09-08 04:16:04,607 - INFO - svc-monitor:backup
2025-09-08 04:16:04,611 - INFO - device-manager:backup
2025-09-08 04:16:04,611 - INFO - config-database
2025-09-08 04:16:04,611 - INFO - config-cassandra:active
2025-09-08 04:16:04,611 - INFO - config-zookeeper:active
2025-09-08 04:16:04,611 - INFO - config-rabbitmq:active
2025-09-08 04:16:04,612 - INFO - control
2025-09-08 04:16:04,622 - INFO - control-nodemgr:active
2025-09-08 04:16:04,655 - INFO - control:active
2025-09-08 04:16:04,656 - INFO - named:active
2025-09-08 04:16:04,667 - INFO - dns:active
2025-09-08 04:16:04,667 - INFO - analytics
2025-09-08 04:16:04,676 - INFO - analytics-nodemgr:active
2025-09-08 04:16:04,691 - INFO - analytics-api:active
2025-09-08 04:16:04,710 - INFO - collector:active
2025-09-08 04:16:04,711 - INFO - analytics-database
2025-09-08 04:16:04,711 - INFO - analytics-cassandra:active
2025-09-08 04:16:04,722 - INFO - analyticsdb-nodemgr:active
2025-09-08 04:16:04,747 - INFO - query-engine:active
2025-09-08 04:16:04,747 - INFO - webui
2025-09-08 04:16:04,748 - INFO - webui:active
2025-09-08 04:16:04,748 - INFO - webui-middleware:active
2025-09-08 04:16:04,748 - INFO - redis:active
2025-09-08 04:16:04,749 - INFO - analytics_snmp
2025-09-08 04:16:04,753 - INFO - snmp-collector:initializing (Database:RabbitMQ[] connection down)
2025-09-08 04:16:04,754 - INFO - snmp-topology:active
2025-09-08 04:16:04,755 - INFO - analytics_alarm
2025-09-08 04:16:04,756 - INFO - alarmgen:active
2025-09-08 04:16:04,867 - INFO - 10.0.0.27
2025-09-08 04:16:04,867 - INFO - config
2025-09-08 04:16:04,875 - INFO - config-nodemgr:active
2025-09-08 04:16:04,885 - INFO - api-server:active
2025-09-08 04:16:04,891 - INFO - schema:active
2025-09-08 04:16:04,897 - INFO - svc-monitor:active
2025-09-08 04:16:04,902 - INFO - device-manager:active
2025-09-08 04:16:04,902 - INFO - config-database
2025-09-08 04:16:04,902 - INFO - config-cassandra:active
2025-09-08 04:16:04,902 - INFO - config-zookeeper:active
2025-09-08 04:16:04,902 - INFO - config-rabbitmq:active
2025-09-08 04:16:04,902 - INFO - control
2025-09-08 04:16:04,907 - INFO - control-nodemgr:active
2025-09-08 04:16:04,913 - INFO - control:active
2025-09-08 04:16:04,913 - INFO - named:active
2025-09-08 04:16:04,919 - INFO - dns:active
2025-09-08 04:16:04,919 - INFO - analytics
2025-09-08 04:16:04,926 - INFO - analytics-nodemgr:active
2025-09-08 04:16:04,931 - INFO - analytics-api:active
2025-09-08 04:16:04,936 - INFO - collector:active
2025-09-08 04:16:04,936 - INFO - analytics-database
2025-09-08 04:16:04,936 - INFO - analytics-cassandra:active
2025-09-08 04:16:04,941 - INFO - analyticsdb-nodemgr:active
2025-09-08 04:16:04,947 - INFO - query-engine:active
2025-09-08 04:16:04,947 - INFO - webui
2025-09-08 04:16:04,947 - INFO - webui:active
2025-09-08 04:16:04,947 - INFO - webui-middleware:active
2025-09-08 04:16:04,947 - INFO - redis:active
2025-09-08 04:16:04,947 - INFO - analytics_snmp
2025-09-08 04:16:04,951 - INFO - snmp-collector:active
2025-09-08 04:16:04,952 - INFO - snmp-topology:active
2025-09-08 04:16:04,952 - INFO - analytics_alarm
2025-09-08 04:16:04,952 - INFO - alarmgen:active
2025-09-08 04:16:05,052 - INFO - 10.0.0.38
2025-09-08 04:16:05,052 - INFO - config
2025-09-08 04:16:05,058 - INFO - config-nodemgr:active
2025-09-08 04:16:05,064 - INFO - api-server:active
2025-09-08 04:16:05,067 - INFO - schema:backup
2025-09-08 04:16:05,071 - INFO - svc-monitor:backup
2025-09-08 04:16:05,074 - INFO - device-manager:backup
2025-09-08 04:16:05,074 - INFO - config-database
2025-09-08 04:16:05,074 - INFO - config-cassandra:active
2025-09-08 04:16:05,074 - INFO - config-zookeeper:active
2025-09-08 04:16:05,074 - INFO - config-rabbitmq:active
2025-09-08 04:16:05,075 - INFO - control
2025-09-08 04:16:05,080 - INFO - control-nodemgr:active
2025-09-08 04:16:05,085 - INFO - control:active
2025-09-08 04:16:05,085 - INFO - named:active
2025-09-08 04:16:05,091 - INFO - dns:active
2025-09-08 04:16:05,091 - INFO - analytics
2025-09-08 04:16:05,097 - INFO - analytics-nodemgr:active
2025-09-08 04:16:05,101 - INFO - analytics-api:active
2025-09-08 04:16:05,107 - INFO - collector:active
2025-09-08 04:16:05,107 - INFO - analytics-database
2025-09-08 04:16:05,107 - INFO - analytics-cassandra:active
2025-09-08 04:16:05,113 - INFO - analyticsdb-nodemgr:active
2025-09-08 04:16:05,119 - INFO - query-engine:active
2025-09-08 04:16:05,119 - INFO - webui
2025-09-08 04:16:05,119 - INFO - webui:active
2025-09-08 04:16:05,120 - INFO - webui-middleware:active
2025-09-08 04:16:05,120 - INFO - redis:active
2025-09-08 04:16:05,120 - INFO - analytics_snmp
2025-09-08 04:16:05,125 - INFO - snmp-collector:active
2025-09-08 04:16:05,125 - INFO - snmp-topology:active
2025-09-08 04:16:05,125 - INFO - analytics_alarm
2025-09-08 04:16:05,125 - INFO - alarmgen:active
2025-09-08 04:16:05,301 - INFO - 10.0.0.65
2025-09-08 04:16:05,302 - INFO - vrouter
2025-09-08 04:16:05,307 - INFO - vrouter-nodemgr:active
2025-09-08 04:16:05,314 - INFO - agent:active
2025-09-08 04:16:05,446 - INFO - 10.0.0.23
2025-09-08 04:16:05,446 - INFO - vrouter
2025-09-08 04:16:05,453 - INFO - vrouter-nodemgr:active
2025-09-08 04:16:05,460 - INFO - agent:active
2025-09-08 04:16:05,460 - DEBUG - defaultdict(, {'10.0.0.41': {'snmp-collector': {'status': 'initializing', 'description': 'Database:RabbitMQ[] connection down'}}})
2025-09-08 04:16:05,460 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 0
2025-09-08 04:16:15,584 - INFO - 10.0.0.41
2025-09-08 04:16:15,584 - INFO - config
2025-09-08 04:16:15,589 - INFO - config-nodemgr:active
2025-09-08 04:16:15,594 - INFO - api-server:active
2025-09-08 04:16:15,598 - INFO - schema:backup
2025-09-08 04:16:15,601 - INFO - svc-monitor:backup
2025-09-08 04:16:15,603 - INFO - device-manager:backup
2025-09-08 04:16:15,603 - INFO - config-database
2025-09-08 04:16:15,604 - INFO - config-cassandra:active
2025-09-08 04:16:15,604 - INFO - config-zookeeper:active
2025-09-08 04:16:15,604 - INFO - config-rabbitmq:active
2025-09-08 04:16:15,604 - INFO - control
2025-09-08 04:16:15,611 - INFO - control-nodemgr:active
2025-09-08 04:16:15,617 - INFO - control:active
2025-09-08 04:16:15,617 - INFO - named:active
2025-09-08 04:16:15,622 - INFO - dns:active
2025-09-08 04:16:15,622 - INFO - analytics
2025-09-08 04:16:15,627 - INFO - analytics-nodemgr:active
2025-09-08 04:16:15,631 - INFO - analytics-api:active
2025-09-08 04:16:15,635 - INFO - collector:active
2025-09-08 04:16:15,635 - INFO - analytics-database
2025-09-08 04:16:15,636 - INFO - analytics-cassandra:active
2025-09-08 04:16:15,640 - INFO - analyticsdb-nodemgr:active
2025-09-08 04:16:15,644 - INFO - query-engine:active
2025-09-08 04:16:15,644 - INFO - webui
2025-09-08 04:16:15,644 - INFO - webui:active
2025-09-08 04:16:15,644 - INFO - webui-middleware:active
2025-09-08 04:16:15,644 - INFO - redis:active
2025-09-08 04:16:15,645 - INFO - analytics_snmp
2025-09-08 04:16:15,648 - INFO - snmp-collector:active
2025-09-08 04:16:15,648 - INFO - snmp-topology:active
2025-09-08 04:16:15,648 - INFO - analytics_alarm
2025-09-08 04:16:15,649 - INFO - alarmgen:active
2025-09-08 04:16:15,749 - INFO - 10.0.0.27
2025-09-08 04:16:15,750 - INFO - config
2025-09-08 04:16:15,756 - INFO - config-nodemgr:active
2025-09-08 04:16:15,761 - INFO - api-server:active
2025-09-08 04:16:15,765 - INFO - schema:active
2025-09-08 04:16:15,769 - INFO - svc-monitor:active
2025-09-08 04:16:15,774 - INFO - device-manager:active
2025-09-08 04:16:15,774 - INFO - config-database
2025-09-08 04:16:15,774 - INFO - config-cassandra:active
2025-09-08 04:16:15,774 - INFO - config-zookeeper:active
2025-09-08 04:16:15,774 - INFO - config-rabbitmq:active
2025-09-08 04:16:15,774 - INFO - control
2025-09-08 04:16:15,779 - INFO - control-nodemgr:active
2025-09-08 04:16:15,785 - INFO - control:active
2025-09-08 04:16:15,785 - INFO - named:active
2025-09-08 04:16:15,790 - INFO - dns:active
2025-09-08 04:16:15,791 - INFO - analytics
2025-09-08 04:16:15,795 - INFO - analytics-nodemgr:active
2025-09-08 04:16:15,799 - INFO - analytics-api:active
2025-09-08 04:16:15,803 - INFO - collector:active
2025-09-08 04:16:15,803 - INFO - analytics-database
2025-09-08 04:16:15,803 - INFO - analytics-cassandra:active
2025-09-08 04:16:15,808 - INFO - analyticsdb-nodemgr:active
2025-09-08 04:16:15,813 - INFO - query-engine:active
2025-09-08 04:16:15,813 - INFO - webui
2025-09-08 04:16:15,813 - INFO - webui:active
2025-09-08 04:16:15,813 - INFO - webui-middleware:active
2025-09-08 04:16:15,813 - INFO - redis:active
2025-09-08 04:16:15,813 - INFO - analytics_snmp
2025-09-08 04:16:15,817 - INFO - snmp-collector:active
2025-09-08 04:16:15,817 - INFO - snmp-topology:active
2025-09-08 04:16:15,817 - INFO - analytics_alarm
2025-09-08 04:16:15,817 - INFO - alarmgen:active
2025-09-08 04:16:15,918 - INFO - 10.0.0.38
2025-09-08 04:16:15,918 - INFO - config
2025-09-08 04:16:15,924 - INFO - config-nodemgr:active
2025-09-08 04:16:15,928 - INFO - api-server:active
2025-09-08 04:16:15,932 - INFO - schema:backup
2025-09-08 04:16:15,937 - INFO - svc-monitor:backup
2025-09-08 04:16:15,943 - INFO - device-manager:backup
2025-09-08 04:16:15,943 - INFO - config-database
2025-09-08 04:16:15,943 - INFO - config-cassandra:active
2025-09-08 04:16:15,943 - INFO - config-zookeeper:active
2025-09-08 04:16:15,943 - INFO - config-rabbitmq:active
2025-09-08 04:16:15,943 - INFO - control
2025-09-08 04:16:15,949 - INFO - control-nodemgr:active
2025-09-08 04:16:15,954 - INFO - control:active
2025-09-08 04:16:15,955 - INFO - named:active
2025-09-08 04:16:15,960 - INFO - dns:active
2025-09-08 04:16:15,960 - INFO - analytics
2025-09-08 04:16:15,965 - INFO - analytics-nodemgr:active
2025-09-08 04:16:15,970 - INFO - analytics-api:active
2025-09-08 04:16:15,977 - INFO - collector:active
2025-09-08 04:16:15,977 - INFO - analytics-database
2025-09-08 04:16:15,977 - INFO - analytics-cassandra:active
2025-09-08 04:16:15,983 - INFO - analyticsdb-nodemgr:active
2025-09-08 04:16:15,990 - INFO - query-engine:active
2025-09-08 04:16:15,990 - INFO - webui
2025-09-08 04:16:15,990 - INFO - webui:active
2025-09-08 04:16:15,991 - INFO - webui-middleware:active
2025-09-08 04:16:15,991 - INFO - redis:active
2025-09-08 04:16:15,991 - INFO - analytics_snmp
2025-09-08 04:16:15,998 - INFO - snmp-collector:active
2025-09-08 04:16:15,998 - INFO - snmp-topology:active
2025-09-08 04:16:15,998 - INFO - analytics_alarm
2025-09-08 04:16:15,998 - INFO - alarmgen:active
2025-09-08 04:16:16,120 - INFO - 10.0.0.65
2025-09-08 04:16:16,120 - INFO - vrouter
2025-09-08 04:16:16,132 - INFO - vrouter-nodemgr:active
2025-09-08 04:16:16,139 - INFO - agent:active
2025-09-08 04:16:16,282 - INFO - 10.0.0.23
2025-09-08 04:16:16,282 - INFO - vrouter
2025-09-08 04:16:16,288 - INFO - vrouter-nodemgr:active
2025-09-08 04:16:16,298 - INFO - agent:active
2025-09-08 04:16:16,299 - INFO - All the contrail services are up on all nodes
2025-09-08 04:16:16,299 - INFO - contrail-status passed
2025-09-08 04:16:17,667 - DEBUG - Skipping xmpp flap check
2025-09-08 04:16:17,667 - INFO - END TEST : test_contrail_status : PASSED[0:00:15]
2025-09-08 04:16:17,667 - INFO - --------------------------------------------------------------------------------