2026-01-26 02:12:55,856 - INFO - Using existing project ['default-domain', 'admin'](7266368c-51f7-4fd8-bfd5-b69aace47550) 2026-01-26 02:12:55,865 - INFO - ================================================================================ 2026-01-26 02:12:55,866 - INFO - STARTING TEST : test_contrail_status 2026-01-26 02:12:55,866 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active 2026-01-26 02:12:56,492 - DEBUG - Nothing to compare xmpp stats {'10.0.0.65': {'10.20.0.129': '0', '10.20.0.25': '0'}, '10.0.0.130': {'10.20.0.25': '0', '10.20.0.129': '0'}} with 2026-01-26 02:12:56,492 - INFO - Initial checks done. Running the testcase now 2026-01-26 02:12:56,493 - INFO - 2026-01-26 02:12:56,571 - INFO - 10.0.0.141 2026-01-26 02:12:56,571 - INFO - config 2026-01-26 02:12:56,577 - INFO - config-nodemgr:active 2026-01-26 02:12:56,581 - INFO - api-server:initializing (Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.141:5000/v3/domains? at UTC 2026-01-26 02:12:36.311409] connection down) 2026-01-26 02:12:56,584 - INFO - schema:active 2026-01-26 02:12:56,588 - INFO - svc-monitor:active 2026-01-26 02:12:56,590 - INFO - device-manager:backup 2026-01-26 02:12:56,591 - INFO - config-database 2026-01-26 02:12:56,591 - INFO - config-cassandra:active 2026-01-26 02:12:56,591 - INFO - config-zookeeper:active 2026-01-26 02:12:56,591 - INFO - config-rabbitmq:active 2026-01-26 02:12:56,591 - INFO - control 2026-01-26 02:12:56,595 - INFO - control-nodemgr:active 2026-01-26 02:12:56,600 - INFO - control:active 2026-01-26 02:12:56,600 - INFO - named:active 2026-01-26 02:12:56,606 - INFO - dns:active 2026-01-26 02:12:56,606 - INFO - analytics 2026-01-26 02:12:56,610 - INFO - analytics-nodemgr:active 2026-01-26 02:12:56,614 - INFO - analytics-api:active 2026-01-26 02:12:56,617 - INFO - collector:active 2026-01-26 02:12:56,617 - INFO - analytics-database 2026-01-26 02:12:56,618 - INFO - analytics-cassandra:active 2026-01-26 02:12:56,621 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:12:56,626 - INFO - query-engine:active 2026-01-26 02:12:56,626 - INFO - webui 2026-01-26 02:12:56,626 - INFO - webui:active 2026-01-26 02:12:56,626 - INFO - webui-middleware:active 2026-01-26 02:12:56,626 - INFO - redis:active 2026-01-26 02:12:56,626 - INFO - analytics_snmp 2026-01-26 02:12:56,630 - INFO - snmp-collector:active 2026-01-26 02:12:56,630 - INFO - snmp-topology:active 2026-01-26 02:12:56,630 - INFO - analytics_alarm 2026-01-26 02:12:56,630 - INFO - alarmgen:active 2026-01-26 02:12:56,696 - INFO - 10.0.0.38 2026-01-26 02:12:56,696 - INFO - config 2026-01-26 02:12:56,703 - INFO - config-nodemgr:active 2026-01-26 02:12:56,717 - INFO - api-server:active 2026-01-26 02:12:56,729 - INFO - schema:backup 2026-01-26 02:12:56,733 - INFO - svc-monitor:backup 2026-01-26 02:12:56,738 - INFO - device-manager:active 2026-01-26 02:12:56,738 - INFO - config-database 2026-01-26 02:12:56,738 - INFO - config-cassandra:active 2026-01-26 02:12:56,738 - INFO - config-zookeeper:active 2026-01-26 02:12:56,738 - INFO - config-rabbitmq:active 2026-01-26 02:12:56,739 - INFO - control 2026-01-26 02:12:56,744 - INFO - control-nodemgr:active 2026-01-26 02:12:56,750 - INFO - control:active 2026-01-26 02:12:56,750 - INFO - named:active 2026-01-26 02:12:56,756 - INFO - dns:active 2026-01-26 02:12:56,756 - INFO - analytics 2026-01-26 02:12:56,762 - INFO - analytics-nodemgr:active 2026-01-26 02:12:56,767 - INFO - analytics-api:active 2026-01-26 02:12:56,772 - INFO - collector:active 2026-01-26 02:12:56,772 - INFO - analytics-database 2026-01-26 02:12:56,772 - INFO - analytics-cassandra:active 2026-01-26 02:12:56,777 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:12:56,782 - INFO - query-engine:active 2026-01-26 02:12:56,782 - INFO - webui 2026-01-26 02:12:56,782 - INFO - webui:active 2026-01-26 02:12:56,782 - INFO - webui-middleware:active 2026-01-26 02:12:56,782 - INFO - redis:active 2026-01-26 02:12:56,782 - INFO - analytics_snmp 2026-01-26 02:12:56,786 - INFO - snmp-collector:active 2026-01-26 02:12:56,786 - INFO - snmp-topology:active 2026-01-26 02:12:56,786 - INFO - analytics_alarm 2026-01-26 02:12:56,786 - INFO - alarmgen:active 2026-01-26 02:12:56,844 - INFO - 10.0.0.240 2026-01-26 02:12:56,844 - INFO - config 2026-01-26 02:12:56,849 - INFO - config-nodemgr:active 2026-01-26 02:12:56,853 - INFO - api-server:active 2026-01-26 02:12:56,856 - INFO - schema:backup 2026-01-26 02:12:56,859 - INFO - svc-monitor:backup 2026-01-26 02:12:56,862 - INFO - device-manager:backup 2026-01-26 02:12:56,862 - INFO - config-database 2026-01-26 02:12:56,862 - INFO - config-cassandra:active 2026-01-26 02:12:56,862 - INFO - config-zookeeper:active 2026-01-26 02:12:56,862 - INFO - config-rabbitmq:active 2026-01-26 02:12:56,862 - INFO - control 2026-01-26 02:12:56,867 - INFO - control-nodemgr:active 2026-01-26 02:12:56,872 - INFO - control:active 2026-01-26 02:12:56,872 - INFO - named:active 2026-01-26 02:12:56,877 - INFO - dns:active 2026-01-26 02:12:56,877 - INFO - analytics 2026-01-26 02:12:56,881 - INFO - analytics-nodemgr:active 2026-01-26 02:12:56,885 - INFO - analytics-api:active 2026-01-26 02:12:56,890 - INFO - collector:active 2026-01-26 02:12:56,890 - INFO - analytics-database 2026-01-26 02:12:56,890 - INFO - analytics-cassandra:active 2026-01-26 02:12:56,895 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:12:56,901 - INFO - query-engine:active 2026-01-26 02:12:56,901 - INFO - webui 2026-01-26 02:12:56,901 - INFO - webui:active 2026-01-26 02:12:56,901 - INFO - webui-middleware:active 2026-01-26 02:12:56,901 - INFO - redis:active 2026-01-26 02:12:56,901 - INFO - analytics_snmp 2026-01-26 02:12:56,905 - INFO - snmp-collector:active 2026-01-26 02:12:56,905 - INFO - snmp-topology:active 2026-01-26 02:12:56,906 - INFO - analytics_alarm 2026-01-26 02:12:56,906 - INFO - alarmgen:active 2026-01-26 02:12:56,962 - INFO - 10.0.0.65 2026-01-26 02:12:56,962 - INFO - vrouter 2026-01-26 02:12:56,967 - INFO - vrouter-nodemgr:active 2026-01-26 02:12:56,972 - INFO - agent:active 2026-01-26 02:12:57,028 - INFO - 10.0.0.130 2026-01-26 02:12:57,028 - INFO - vrouter 2026-01-26 02:12:57,035 - INFO - vrouter-nodemgr:active 2026-01-26 02:12:57,040 - INFO - agent:active 2026-01-26 02:12:57,040 - DEBUG - defaultdict(, {'10.0.0.141': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.141:5000/v3/domains? at UTC 2026-01-26 02:12:36.311409] connection down'}}}) 2026-01-26 02:12:57,040 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 0 2026-01-26 02:13:07,108 - INFO - 10.0.0.141 2026-01-26 02:13:07,109 - INFO - config 2026-01-26 02:13:07,115 - INFO - config-nodemgr:active 2026-01-26 02:13:07,119 - INFO - api-server:initializing (Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.141:5000/v3/domains? at UTC 2026-01-26 02:12:36.311409] connection down) 2026-01-26 02:13:07,123 - INFO - schema:active 2026-01-26 02:13:07,127 - INFO - svc-monitor:active 2026-01-26 02:13:07,130 - INFO - device-manager:backup 2026-01-26 02:13:07,130 - INFO - config-database 2026-01-26 02:13:07,130 - INFO - config-cassandra:active 2026-01-26 02:13:07,130 - INFO - config-zookeeper:active 2026-01-26 02:13:07,130 - INFO - config-rabbitmq:active 2026-01-26 02:13:07,130 - INFO - control 2026-01-26 02:13:07,134 - INFO - control-nodemgr:active 2026-01-26 02:13:07,139 - INFO - control:active 2026-01-26 02:13:07,139 - INFO - named:active 2026-01-26 02:13:07,146 - INFO - dns:active 2026-01-26 02:13:07,146 - INFO - analytics 2026-01-26 02:13:07,150 - INFO - analytics-nodemgr:active 2026-01-26 02:13:07,153 - INFO - analytics-api:active 2026-01-26 02:13:07,157 - INFO - collector:active 2026-01-26 02:13:07,157 - INFO - analytics-database 2026-01-26 02:13:07,157 - INFO - analytics-cassandra:active 2026-01-26 02:13:07,161 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:13:07,165 - INFO - query-engine:active 2026-01-26 02:13:07,166 - INFO - webui 2026-01-26 02:13:07,166 - INFO - webui:active 2026-01-26 02:13:07,166 - INFO - webui-middleware:active 2026-01-26 02:13:07,166 - INFO - redis:active 2026-01-26 02:13:07,166 - INFO - analytics_snmp 2026-01-26 02:13:07,169 - INFO - snmp-collector:active 2026-01-26 02:13:07,170 - INFO - snmp-topology:active 2026-01-26 02:13:07,170 - INFO - analytics_alarm 2026-01-26 02:13:07,170 - INFO - alarmgen:active 2026-01-26 02:13:07,225 - INFO - 10.0.0.38 2026-01-26 02:13:07,225 - INFO - config 2026-01-26 02:13:07,232 - INFO - config-nodemgr:active 2026-01-26 02:13:07,236 - INFO - api-server:active 2026-01-26 02:13:07,239 - INFO - schema:backup 2026-01-26 02:13:07,242 - INFO - svc-monitor:backup 2026-01-26 02:13:07,247 - INFO - device-manager:active 2026-01-26 02:13:07,247 - INFO - config-database 2026-01-26 02:13:07,247 - INFO - config-cassandra:active 2026-01-26 02:13:07,247 - INFO - config-zookeeper:active 2026-01-26 02:13:07,247 - INFO - config-rabbitmq:active 2026-01-26 02:13:07,247 - INFO - control 2026-01-26 02:13:07,251 - INFO - control-nodemgr:active 2026-01-26 02:13:07,256 - INFO - control:active 2026-01-26 02:13:07,256 - INFO - named:active 2026-01-26 02:13:07,261 - INFO - dns:active 2026-01-26 02:13:07,261 - INFO - analytics 2026-01-26 02:13:07,266 - INFO - analytics-nodemgr:active 2026-01-26 02:13:07,271 - INFO - analytics-api:active 2026-01-26 02:13:07,276 - INFO - collector:active 2026-01-26 02:13:07,277 - INFO - analytics-database 2026-01-26 02:13:07,277 - INFO - analytics-cassandra:active 2026-01-26 02:13:07,282 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:13:07,287 - INFO - query-engine:active 2026-01-26 02:13:07,287 - INFO - webui 2026-01-26 02:13:07,287 - INFO - webui:active 2026-01-26 02:13:07,287 - INFO - webui-middleware:active 2026-01-26 02:13:07,287 - INFO - redis:active 2026-01-26 02:13:07,287 - INFO - analytics_snmp 2026-01-26 02:13:07,291 - INFO - snmp-collector:active 2026-01-26 02:13:07,291 - INFO - snmp-topology:active 2026-01-26 02:13:07,291 - INFO - analytics_alarm 2026-01-26 02:13:07,291 - INFO - alarmgen:active 2026-01-26 02:13:07,348 - INFO - 10.0.0.240 2026-01-26 02:13:07,348 - INFO - config 2026-01-26 02:13:07,354 - INFO - config-nodemgr:active 2026-01-26 02:13:07,358 - INFO - api-server:active 2026-01-26 02:13:07,362 - INFO - schema:backup 2026-01-26 02:13:07,365 - INFO - svc-monitor:backup 2026-01-26 02:13:07,367 - INFO - device-manager:backup 2026-01-26 02:13:07,368 - INFO - config-database 2026-01-26 02:13:07,368 - INFO - config-cassandra:active 2026-01-26 02:13:07,368 - INFO - config-zookeeper:active 2026-01-26 02:13:07,368 - INFO - config-rabbitmq:active 2026-01-26 02:13:07,368 - INFO - control 2026-01-26 02:13:07,372 - INFO - control-nodemgr:active 2026-01-26 02:13:07,377 - INFO - control:active 2026-01-26 02:13:07,377 - INFO - named:active 2026-01-26 02:13:07,381 - INFO - dns:active 2026-01-26 02:13:07,381 - INFO - analytics 2026-01-26 02:13:07,386 - INFO - analytics-nodemgr:active 2026-01-26 02:13:07,391 - INFO - analytics-api:active 2026-01-26 02:13:07,396 - INFO - collector:active 2026-01-26 02:13:07,396 - INFO - analytics-database 2026-01-26 02:13:07,397 - INFO - analytics-cassandra:active 2026-01-26 02:13:07,401 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:13:07,406 - INFO - query-engine:active 2026-01-26 02:13:07,407 - INFO - webui 2026-01-26 02:13:07,407 - INFO - webui:active 2026-01-26 02:13:07,407 - INFO - webui-middleware:active 2026-01-26 02:13:07,407 - INFO - redis:active 2026-01-26 02:13:07,407 - INFO - analytics_snmp 2026-01-26 02:13:07,411 - INFO - snmp-collector:active 2026-01-26 02:13:07,412 - INFO - snmp-topology:active 2026-01-26 02:13:07,412 - INFO - analytics_alarm 2026-01-26 02:13:07,412 - INFO - alarmgen:active 2026-01-26 02:13:07,457 - INFO - 10.0.0.65 2026-01-26 02:13:07,457 - INFO - vrouter 2026-01-26 02:13:07,462 - INFO - vrouter-nodemgr:active 2026-01-26 02:13:07,467 - INFO - agent:active 2026-01-26 02:13:07,514 - INFO - 10.0.0.130 2026-01-26 02:13:07,514 - INFO - vrouter 2026-01-26 02:13:07,520 - INFO - vrouter-nodemgr:active 2026-01-26 02:13:07,526 - INFO - agent:active 2026-01-26 02:13:07,527 - DEBUG - defaultdict(, {'10.0.0.141': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.141:5000/v3/domains? at UTC 2026-01-26 02:12:36.311409] connection down'}}}) 2026-01-26 02:13:07,527 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 1 2026-01-26 02:13:17,582 - INFO - 10.0.0.141 2026-01-26 02:13:17,582 - INFO - config 2026-01-26 02:13:17,588 - INFO - config-nodemgr:active 2026-01-26 02:13:17,592 - INFO - api-server:initializing (Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.141:5000/v3/domains? at UTC 2026-01-26 02:12:36.311409] connection down) 2026-01-26 02:13:17,597 - INFO - schema:active 2026-01-26 02:13:17,601 - INFO - svc-monitor:active 2026-01-26 02:13:17,603 - INFO - device-manager:backup 2026-01-26 02:13:17,604 - INFO - config-database 2026-01-26 02:13:17,604 - INFO - config-cassandra:active 2026-01-26 02:13:17,604 - INFO - config-zookeeper:active 2026-01-26 02:13:17,604 - INFO - config-rabbitmq:active 2026-01-26 02:13:17,604 - INFO - control 2026-01-26 02:13:17,610 - INFO - control-nodemgr:active 2026-01-26 02:13:17,617 - INFO - control:active 2026-01-26 02:13:17,617 - INFO - named:active 2026-01-26 02:13:17,624 - INFO - dns:active 2026-01-26 02:13:17,624 - INFO - analytics 2026-01-26 02:13:17,628 - INFO - analytics-nodemgr:active 2026-01-26 02:13:17,632 - INFO - analytics-api:active 2026-01-26 02:13:17,635 - INFO - collector:active 2026-01-26 02:13:17,636 - INFO - analytics-database 2026-01-26 02:13:17,636 - INFO - analytics-cassandra:active 2026-01-26 02:13:17,639 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:13:17,644 - INFO - query-engine:active 2026-01-26 02:13:17,644 - INFO - webui 2026-01-26 02:13:17,644 - INFO - webui:active 2026-01-26 02:13:17,644 - INFO - webui-middleware:active 2026-01-26 02:13:17,644 - INFO - redis:active 2026-01-26 02:13:17,644 - INFO - analytics_snmp 2026-01-26 02:13:17,648 - INFO - snmp-collector:active 2026-01-26 02:13:17,648 - INFO - snmp-topology:active 2026-01-26 02:13:17,648 - INFO - analytics_alarm 2026-01-26 02:13:17,648 - INFO - alarmgen:active 2026-01-26 02:13:17,705 - INFO - 10.0.0.38 2026-01-26 02:13:17,705 - INFO - config 2026-01-26 02:13:17,711 - INFO - config-nodemgr:active 2026-01-26 02:13:17,717 - INFO - api-server:active 2026-01-26 02:13:17,721 - INFO - schema:backup 2026-01-26 02:13:17,725 - INFO - svc-monitor:backup 2026-01-26 02:13:17,729 - INFO - device-manager:active 2026-01-26 02:13:17,729 - INFO - config-database 2026-01-26 02:13:17,729 - INFO - config-cassandra:active 2026-01-26 02:13:17,729 - INFO - config-zookeeper:active 2026-01-26 02:13:17,729 - INFO - config-rabbitmq:active 2026-01-26 02:13:17,729 - INFO - control 2026-01-26 02:13:17,736 - INFO - control-nodemgr:active 2026-01-26 02:13:17,743 - INFO - control:active 2026-01-26 02:13:17,743 - INFO - named:active 2026-01-26 02:13:17,749 - INFO - dns:active 2026-01-26 02:13:17,749 - INFO - analytics 2026-01-26 02:13:17,754 - INFO - analytics-nodemgr:active 2026-01-26 02:13:17,758 - INFO - analytics-api:active 2026-01-26 02:13:17,763 - INFO - collector:active 2026-01-26 02:13:17,763 - INFO - analytics-database 2026-01-26 02:13:17,763 - INFO - analytics-cassandra:active 2026-01-26 02:13:17,768 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:13:17,773 - INFO - query-engine:active 2026-01-26 02:13:17,773 - INFO - webui 2026-01-26 02:13:17,773 - INFO - webui:active 2026-01-26 02:13:17,773 - INFO - webui-middleware:active 2026-01-26 02:13:17,773 - INFO - redis:active 2026-01-26 02:13:17,774 - INFO - analytics_snmp 2026-01-26 02:13:17,778 - INFO - snmp-collector:active 2026-01-26 02:13:17,778 - INFO - snmp-topology:active 2026-01-26 02:13:17,778 - INFO - analytics_alarm 2026-01-26 02:13:17,778 - INFO - alarmgen:active 2026-01-26 02:13:17,834 - INFO - 10.0.0.240 2026-01-26 02:13:17,834 - INFO - config 2026-01-26 02:13:17,845 - INFO - config-nodemgr:active 2026-01-26 02:13:17,850 - INFO - api-server:active 2026-01-26 02:13:17,854 - INFO - schema:backup 2026-01-26 02:13:17,858 - INFO - svc-monitor:backup 2026-01-26 02:13:17,862 - INFO - device-manager:backup 2026-01-26 02:13:17,862 - INFO - config-database 2026-01-26 02:13:17,862 - INFO - config-cassandra:active 2026-01-26 02:13:17,862 - INFO - config-zookeeper:active 2026-01-26 02:13:17,862 - INFO - config-rabbitmq:active 2026-01-26 02:13:17,862 - INFO - control 2026-01-26 02:13:17,867 - INFO - control-nodemgr:active 2026-01-26 02:13:17,874 - INFO - control:active 2026-01-26 02:13:17,877 - INFO - named:active 2026-01-26 02:13:17,882 - INFO - dns:active 2026-01-26 02:13:17,882 - INFO - analytics 2026-01-26 02:13:17,887 - INFO - analytics-nodemgr:active 2026-01-26 02:13:17,892 - INFO - analytics-api:active 2026-01-26 02:13:17,896 - INFO - collector:active 2026-01-26 02:13:17,897 - INFO - analytics-database 2026-01-26 02:13:17,897 - INFO - analytics-cassandra:active 2026-01-26 02:13:17,901 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:13:17,907 - INFO - query-engine:active 2026-01-26 02:13:17,907 - INFO - webui 2026-01-26 02:13:17,907 - INFO - webui:active 2026-01-26 02:13:17,907 - INFO - webui-middleware:active 2026-01-26 02:13:17,907 - INFO - redis:active 2026-01-26 02:13:17,907 - INFO - analytics_snmp 2026-01-26 02:13:17,912 - INFO - snmp-collector:active 2026-01-26 02:13:17,912 - INFO - snmp-topology:active 2026-01-26 02:13:17,912 - INFO - analytics_alarm 2026-01-26 02:13:17,912 - INFO - alarmgen:active 2026-01-26 02:13:17,959 - INFO - 10.0.0.65 2026-01-26 02:13:17,959 - INFO - vrouter 2026-01-26 02:13:17,964 - INFO - vrouter-nodemgr:active 2026-01-26 02:13:17,970 - INFO - agent:active 2026-01-26 02:13:18,027 - INFO - 10.0.0.130 2026-01-26 02:13:18,027 - INFO - vrouter 2026-01-26 02:13:18,033 - INFO - vrouter-nodemgr:active 2026-01-26 02:13:18,038 - INFO - agent:active 2026-01-26 02:13:18,038 - DEBUG - defaultdict(, {'10.0.0.141': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.141:5000/v3/domains? at UTC 2026-01-26 02:12:36.311409] connection down'}}}) 2026-01-26 02:13:18,038 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 2 2026-01-26 02:13:28,125 - INFO - 10.0.0.141 2026-01-26 02:13:28,125 - INFO - config 2026-01-26 02:13:28,130 - INFO - config-nodemgr:active 2026-01-26 02:13:28,135 - INFO - api-server:initializing (Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.141:5000/v3/domains? at UTC 2026-01-26 02:12:36.311409] connection down) 2026-01-26 02:13:28,144 - INFO - schema:active 2026-01-26 02:13:28,148 - INFO - svc-monitor:active 2026-01-26 02:13:28,151 - INFO - device-manager:backup 2026-01-26 02:13:28,151 - INFO - config-database 2026-01-26 02:13:28,151 - INFO - config-cassandra:active 2026-01-26 02:13:28,151 - INFO - config-zookeeper:active 2026-01-26 02:13:28,151 - INFO - config-rabbitmq:active 2026-01-26 02:13:28,151 - INFO - control 2026-01-26 02:13:28,156 - INFO - control-nodemgr:active 2026-01-26 02:13:28,160 - INFO - control:active 2026-01-26 02:13:28,160 - INFO - named:active 2026-01-26 02:13:28,165 - INFO - dns:active 2026-01-26 02:13:28,165 - INFO - analytics 2026-01-26 02:13:28,169 - INFO - analytics-nodemgr:active 2026-01-26 02:13:28,173 - INFO - analytics-api:active 2026-01-26 02:13:28,177 - INFO - collector:active 2026-01-26 02:13:28,177 - INFO - analytics-database 2026-01-26 02:13:28,177 - INFO - analytics-cassandra:active 2026-01-26 02:13:28,181 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:13:28,186 - INFO - query-engine:active 2026-01-26 02:13:28,186 - INFO - webui 2026-01-26 02:13:28,186 - INFO - webui:active 2026-01-26 02:13:28,186 - INFO - webui-middleware:active 2026-01-26 02:13:28,186 - INFO - redis:active 2026-01-26 02:13:28,186 - INFO - analytics_snmp 2026-01-26 02:13:28,189 - INFO - snmp-collector:active 2026-01-26 02:13:28,189 - INFO - snmp-topology:active 2026-01-26 02:13:28,189 - INFO - analytics_alarm 2026-01-26 02:13:28,189 - INFO - alarmgen:active 2026-01-26 02:13:28,244 - INFO - 10.0.0.38 2026-01-26 02:13:28,244 - INFO - config 2026-01-26 02:13:28,249 - INFO - config-nodemgr:active 2026-01-26 02:13:28,253 - INFO - api-server:active 2026-01-26 02:13:28,256 - INFO - schema:backup 2026-01-26 02:13:28,259 - INFO - svc-monitor:backup 2026-01-26 02:13:28,262 - INFO - device-manager:active 2026-01-26 02:13:28,262 - INFO - config-database 2026-01-26 02:13:28,262 - INFO - config-cassandra:active 2026-01-26 02:13:28,262 - INFO - config-zookeeper:active 2026-01-26 02:13:28,262 - INFO - config-rabbitmq:active 2026-01-26 02:13:28,263 - INFO - control 2026-01-26 02:13:28,268 - INFO - control-nodemgr:active 2026-01-26 02:13:28,273 - INFO - control:active 2026-01-26 02:13:28,273 - INFO - named:active 2026-01-26 02:13:28,278 - INFO - dns:active 2026-01-26 02:13:28,278 - INFO - analytics 2026-01-26 02:13:28,282 - INFO - analytics-nodemgr:active 2026-01-26 02:13:28,287 - INFO - analytics-api:active 2026-01-26 02:13:28,292 - INFO - collector:active 2026-01-26 02:13:28,293 - INFO - analytics-database 2026-01-26 02:13:28,293 - INFO - analytics-cassandra:active 2026-01-26 02:13:28,298 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:13:28,303 - INFO - query-engine:active 2026-01-26 02:13:28,303 - INFO - webui 2026-01-26 02:13:28,304 - INFO - webui:active 2026-01-26 02:13:28,304 - INFO - webui-middleware:active 2026-01-26 02:13:28,304 - INFO - redis:active 2026-01-26 02:13:28,304 - INFO - analytics_snmp 2026-01-26 02:13:28,309 - INFO - snmp-collector:active 2026-01-26 02:13:28,309 - INFO - snmp-topology:active 2026-01-26 02:13:28,310 - INFO - analytics_alarm 2026-01-26 02:13:28,310 - INFO - alarmgen:active 2026-01-26 02:13:28,366 - INFO - 10.0.0.240 2026-01-26 02:13:28,366 - INFO - config 2026-01-26 02:13:28,371 - INFO - config-nodemgr:active 2026-01-26 02:13:28,375 - INFO - api-server:active 2026-01-26 02:13:28,378 - INFO - schema:backup 2026-01-26 02:13:28,380 - INFO - svc-monitor:backup 2026-01-26 02:13:28,383 - INFO - device-manager:backup 2026-01-26 02:13:28,383 - INFO - config-database 2026-01-26 02:13:28,383 - INFO - config-cassandra:active 2026-01-26 02:13:28,384 - INFO - config-zookeeper:active 2026-01-26 02:13:28,384 - INFO - config-rabbitmq:active 2026-01-26 02:13:28,384 - INFO - control 2026-01-26 02:13:28,389 - INFO - control-nodemgr:active 2026-01-26 02:13:28,394 - INFO - control:active 2026-01-26 02:13:28,394 - INFO - named:active 2026-01-26 02:13:28,401 - INFO - dns:active 2026-01-26 02:13:28,401 - INFO - analytics 2026-01-26 02:13:28,406 - INFO - analytics-nodemgr:active 2026-01-26 02:13:28,411 - INFO - analytics-api:active 2026-01-26 02:13:28,415 - INFO - collector:active 2026-01-26 02:13:28,416 - INFO - analytics-database 2026-01-26 02:13:28,416 - INFO - analytics-cassandra:active 2026-01-26 02:13:28,420 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:13:28,426 - INFO - query-engine:active 2026-01-26 02:13:28,426 - INFO - webui 2026-01-26 02:13:28,426 - INFO - webui:active 2026-01-26 02:13:28,426 - INFO - webui-middleware:active 2026-01-26 02:13:28,426 - INFO - redis:active 2026-01-26 02:13:28,427 - INFO - analytics_snmp 2026-01-26 02:13:28,431 - INFO - snmp-collector:active 2026-01-26 02:13:28,431 - INFO - snmp-topology:active 2026-01-26 02:13:28,431 - INFO - analytics_alarm 2026-01-26 02:13:28,431 - INFO - alarmgen:active 2026-01-26 02:13:28,475 - INFO - 10.0.0.65 2026-01-26 02:13:28,475 - INFO - vrouter 2026-01-26 02:13:28,480 - INFO - vrouter-nodemgr:active 2026-01-26 02:13:28,486 - INFO - agent:active 2026-01-26 02:13:28,531 - INFO - 10.0.0.130 2026-01-26 02:13:28,532 - INFO - vrouter 2026-01-26 02:13:28,537 - INFO - vrouter-nodemgr:active 2026-01-26 02:13:28,542 - INFO - agent:active 2026-01-26 02:13:28,542 - DEBUG - defaultdict(, {'10.0.0.141': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.141:5000/v3/domains? at UTC 2026-01-26 02:12:36.311409] connection down'}}}) 2026-01-26 02:13:28,542 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 3 2026-01-26 02:13:38,600 - INFO - 10.0.0.141 2026-01-26 02:13:38,600 - INFO - config 2026-01-26 02:13:38,605 - INFO - config-nodemgr:active 2026-01-26 02:13:38,608 - INFO - api-server:active 2026-01-26 02:13:38,612 - INFO - schema:active 2026-01-26 02:13:38,615 - INFO - svc-monitor:active 2026-01-26 02:13:38,618 - INFO - device-manager:backup 2026-01-26 02:13:38,618 - INFO - config-database 2026-01-26 02:13:38,618 - INFO - config-cassandra:active 2026-01-26 02:13:38,618 - INFO - config-zookeeper:active 2026-01-26 02:13:38,618 - INFO - config-rabbitmq:active 2026-01-26 02:13:38,618 - INFO - control 2026-01-26 02:13:38,622 - INFO - control-nodemgr:active 2026-01-26 02:13:38,626 - INFO - control:active 2026-01-26 02:13:38,626 - INFO - named:active 2026-01-26 02:13:38,631 - INFO - dns:active 2026-01-26 02:13:38,631 - INFO - analytics 2026-01-26 02:13:38,636 - INFO - analytics-nodemgr:active 2026-01-26 02:13:38,639 - INFO - analytics-api:active 2026-01-26 02:13:38,644 - INFO - collector:active 2026-01-26 02:13:38,644 - INFO - analytics-database 2026-01-26 02:13:38,644 - INFO - analytics-cassandra:active 2026-01-26 02:13:38,648 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:13:38,654 - INFO - query-engine:active 2026-01-26 02:13:38,654 - INFO - webui 2026-01-26 02:13:38,654 - INFO - webui:active 2026-01-26 02:13:38,654 - INFO - webui-middleware:active 2026-01-26 02:13:38,654 - INFO - redis:active 2026-01-26 02:13:38,654 - INFO - analytics_snmp 2026-01-26 02:13:38,658 - INFO - snmp-collector:active 2026-01-26 02:13:38,658 - INFO - snmp-topology:active 2026-01-26 02:13:38,658 - INFO - analytics_alarm 2026-01-26 02:13:38,658 - INFO - alarmgen:active 2026-01-26 02:13:38,713 - INFO - 10.0.0.38 2026-01-26 02:13:38,713 - INFO - config 2026-01-26 02:13:38,719 - INFO - config-nodemgr:active 2026-01-26 02:13:38,723 - INFO - api-server:active 2026-01-26 02:13:38,727 - INFO - schema:backup 2026-01-26 02:13:38,730 - INFO - svc-monitor:backup 2026-01-26 02:13:38,734 - INFO - device-manager:active 2026-01-26 02:13:38,734 - INFO - config-database 2026-01-26 02:13:38,734 - INFO - config-cassandra:active 2026-01-26 02:13:38,734 - INFO - config-zookeeper:active 2026-01-26 02:13:38,734 - INFO - config-rabbitmq:active 2026-01-26 02:13:38,735 - INFO - control 2026-01-26 02:13:38,739 - INFO - control-nodemgr:active 2026-01-26 02:13:38,745 - INFO - control:active 2026-01-26 02:13:38,745 - INFO - named:active 2026-01-26 02:13:38,751 - INFO - dns:active 2026-01-26 02:13:38,751 - INFO - analytics 2026-01-26 02:13:38,755 - INFO - analytics-nodemgr:active 2026-01-26 02:13:38,759 - INFO - analytics-api:active 2026-01-26 02:13:38,764 - INFO - collector:active 2026-01-26 02:13:38,764 - INFO - analytics-database 2026-01-26 02:13:38,764 - INFO - analytics-cassandra:active 2026-01-26 02:13:38,768 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:13:38,776 - INFO - query-engine:active 2026-01-26 02:13:38,776 - INFO - webui 2026-01-26 02:13:38,776 - INFO - webui:active 2026-01-26 02:13:38,776 - INFO - webui-middleware:active 2026-01-26 02:13:38,776 - INFO - redis:active 2026-01-26 02:13:38,776 - INFO - analytics_snmp 2026-01-26 02:13:38,780 - INFO - snmp-collector:active 2026-01-26 02:13:38,780 - INFO - snmp-topology:active 2026-01-26 02:13:38,780 - INFO - analytics_alarm 2026-01-26 02:13:38,780 - INFO - alarmgen:active 2026-01-26 02:13:38,825 - INFO - 10.0.0.240 2026-01-26 02:13:38,825 - INFO - config 2026-01-26 02:13:38,830 - INFO - config-nodemgr:active 2026-01-26 02:13:38,834 - INFO - api-server:active 2026-01-26 02:13:38,838 - INFO - schema:backup 2026-01-26 02:13:38,841 - INFO - svc-monitor:backup 2026-01-26 02:13:38,844 - INFO - device-manager:backup 2026-01-26 02:13:38,844 - INFO - config-database 2026-01-26 02:13:38,844 - INFO - config-cassandra:active 2026-01-26 02:13:38,844 - INFO - config-zookeeper:active 2026-01-26 02:13:38,844 - INFO - config-rabbitmq:active 2026-01-26 02:13:38,844 - INFO - control 2026-01-26 02:13:38,849 - INFO - control-nodemgr:active 2026-01-26 02:13:38,854 - INFO - control:active 2026-01-26 02:13:38,854 - INFO - named:active 2026-01-26 02:13:38,858 - INFO - dns:active 2026-01-26 02:13:38,858 - INFO - analytics 2026-01-26 02:13:38,864 - INFO - analytics-nodemgr:active 2026-01-26 02:13:38,868 - INFO - analytics-api:active 2026-01-26 02:13:38,872 - INFO - collector:active 2026-01-26 02:13:38,873 - INFO - analytics-database 2026-01-26 02:13:38,873 - INFO - analytics-cassandra:active 2026-01-26 02:13:38,877 - INFO - analyticsdb-nodemgr:active 2026-01-26 02:13:38,882 - INFO - query-engine:active 2026-01-26 02:13:38,882 - INFO - webui 2026-01-26 02:13:38,882 - INFO - webui:active 2026-01-26 02:13:38,882 - INFO - webui-middleware:active 2026-01-26 02:13:38,882 - INFO - redis:active 2026-01-26 02:13:38,882 - INFO - analytics_snmp 2026-01-26 02:13:38,886 - INFO - snmp-collector:active 2026-01-26 02:13:38,887 - INFO - snmp-topology:active 2026-01-26 02:13:38,887 - INFO - analytics_alarm 2026-01-26 02:13:38,887 - INFO - alarmgen:active 2026-01-26 02:13:38,942 - INFO - 10.0.0.65 2026-01-26 02:13:38,942 - INFO - vrouter 2026-01-26 02:13:38,947 - INFO - vrouter-nodemgr:active 2026-01-26 02:13:38,953 - INFO - agent:active 2026-01-26 02:13:38,998 - INFO - 10.0.0.130 2026-01-26 02:13:38,998 - INFO - vrouter 2026-01-26 02:13:39,003 - INFO - vrouter-nodemgr:active 2026-01-26 02:13:39,009 - INFO - agent:active 2026-01-26 02:13:39,009 - INFO - All the contrail services are up on all nodes 2026-01-26 02:13:39,009 - INFO - contrail-status passed 2026-01-26 02:13:39,663 - DEBUG - No XMPP flaps were noticed during the test
2026-01-26 02:13:39,663 - INFO - END TEST : test_contrail_status : PASSED[0:00:44]
2026-01-26 02:13:39,663 - INFO - -------------------------------------------------------------------------------- 2026-01-26 04:05:55,085 - INFO - Using existing project ['default-domain', 'admin'](7266368c-51f7-4fd8-bfd5-b69aace47550) 2026-01-26 04:05:55,093 - INFO - ================================================================================ 2026-01-26 04:05:55,093 - INFO - STARTING TEST : test_contrail_status 2026-01-26 04:05:55,093 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active 2026-01-26 04:05:56,386 - DEBUG - Skipping xmpp flap check 2026-01-26 04:05:56,386 - INFO - Initial checks done. Running the testcase now 2026-01-26 04:05:56,386 - INFO - 2026-01-26 04:05:56,542 - INFO - 10.0.0.141 2026-01-26 04:05:56,542 - INFO - config 2026-01-26 04:05:56,550 - INFO - config-nodemgr:active 2026-01-26 04:05:56,554 - INFO - api-server:active 2026-01-26 04:05:56,558 - INFO - schema:active 2026-01-26 04:05:56,562 - INFO - svc-monitor:active 2026-01-26 04:05:56,565 - INFO - device-manager:backup 2026-01-26 04:05:56,565 - INFO - config-database 2026-01-26 04:05:56,565 - INFO - config-cassandra:active 2026-01-26 04:05:56,565 - INFO - config-zookeeper:active 2026-01-26 04:05:56,565 - INFO - config-rabbitmq:active 2026-01-26 04:05:56,565 - INFO - control 2026-01-26 04:05:56,569 - INFO - control-nodemgr:active 2026-01-26 04:05:56,575 - INFO - control:active 2026-01-26 04:05:56,575 - INFO - named:active 2026-01-26 04:05:56,586 - INFO - dns:active 2026-01-26 04:05:56,586 - INFO - analytics 2026-01-26 04:05:56,592 - INFO - analytics-nodemgr:active 2026-01-26 04:05:56,596 - INFO - analytics-api:active 2026-01-26 04:05:56,620 - INFO - collector:active 2026-01-26 04:05:56,620 - INFO - analytics-database 2026-01-26 04:05:56,620 - INFO - analytics-cassandra:active 2026-01-26 04:05:56,624 - INFO - analyticsdb-nodemgr:active 2026-01-26 04:05:56,652 - INFO - query-engine:active 2026-01-26 04:05:56,652 - INFO - webui 2026-01-26 04:05:56,652 - INFO - webui:active 2026-01-26 04:05:56,652 - INFO - webui-middleware:active 2026-01-26 04:05:56,652 - INFO - redis:active 2026-01-26 04:05:56,653 - INFO - analytics_snmp 2026-01-26 04:05:56,657 - INFO - snmp-collector:active 2026-01-26 04:05:56,657 - INFO - snmp-topology:active 2026-01-26 04:05:56,657 - INFO - analytics_alarm 2026-01-26 04:05:56,657 - INFO - alarmgen:active 2026-01-26 04:05:56,756 - INFO - 10.0.0.38 2026-01-26 04:05:56,756 - INFO - config 2026-01-26 04:05:56,762 - INFO - config-nodemgr:active 2026-01-26 04:05:56,766 - INFO - api-server:active 2026-01-26 04:05:56,769 - INFO - schema:backup 2026-01-26 04:05:56,772 - INFO - svc-monitor:backup 2026-01-26 04:05:56,777 - INFO - device-manager:active 2026-01-26 04:05:56,777 - INFO - config-database 2026-01-26 04:05:56,777 - INFO - config-cassandra:active 2026-01-26 04:05:56,777 - INFO - config-zookeeper:active 2026-01-26 04:05:56,777 - INFO - config-rabbitmq:active 2026-01-26 04:05:56,778 - INFO - control 2026-01-26 04:05:56,782 - INFO - control-nodemgr:active 2026-01-26 04:05:56,787 - INFO - control:active 2026-01-26 04:05:56,787 - INFO - named:active 2026-01-26 04:05:56,794 - INFO - dns:active 2026-01-26 04:05:56,794 - INFO - analytics 2026-01-26 04:05:56,798 - INFO - analytics-nodemgr:active 2026-01-26 04:05:56,803 - INFO - analytics-api:active 2026-01-26 04:05:56,810 - INFO - collector:active 2026-01-26 04:05:56,810 - INFO - analytics-database 2026-01-26 04:05:56,810 - INFO - analytics-cassandra:active 2026-01-26 04:05:56,815 - INFO - analyticsdb-nodemgr:active 2026-01-26 04:05:56,821 - INFO - query-engine:active 2026-01-26 04:05:56,822 - INFO - webui 2026-01-26 04:05:56,822 - INFO - webui:active 2026-01-26 04:05:56,822 - INFO - webui-middleware:active 2026-01-26 04:05:56,822 - INFO - redis:active 2026-01-26 04:05:56,822 - INFO - analytics_snmp 2026-01-26 04:05:56,826 - INFO - snmp-collector:active 2026-01-26 04:05:56,826 - INFO - snmp-topology:active 2026-01-26 04:05:56,826 - INFO - analytics_alarm 2026-01-26 04:05:56,826 - INFO - alarmgen:active 2026-01-26 04:05:56,928 - INFO - 10.0.0.240 2026-01-26 04:05:56,929 - INFO - config 2026-01-26 04:05:56,935 - INFO - config-nodemgr:active 2026-01-26 04:05:56,942 - INFO - api-server:active 2026-01-26 04:05:56,946 - INFO - schema:backup 2026-01-26 04:05:56,950 - INFO - svc-monitor:backup 2026-01-26 04:05:56,953 - INFO - device-manager:backup 2026-01-26 04:05:56,954 - INFO - config-database 2026-01-26 04:05:56,954 - INFO - config-cassandra:active 2026-01-26 04:05:56,954 - INFO - config-zookeeper:active 2026-01-26 04:05:56,954 - INFO - config-rabbitmq:active 2026-01-26 04:05:56,954 - INFO - control 2026-01-26 04:05:56,959 - INFO - control-nodemgr:active 2026-01-26 04:05:56,965 - INFO - control:active 2026-01-26 04:05:56,965 - INFO - named:active 2026-01-26 04:05:56,990 - INFO - dns:active 2026-01-26 04:05:56,990 - INFO - analytics 2026-01-26 04:05:56,998 - INFO - analytics-nodemgr:active 2026-01-26 04:05:57,003 - INFO - analytics-api:active 2026-01-26 04:05:57,010 - INFO - collector:active 2026-01-26 04:05:57,010 - INFO - analytics-database 2026-01-26 04:05:57,010 - INFO - analytics-cassandra:active 2026-01-26 04:05:57,017 - INFO - analyticsdb-nodemgr:active 2026-01-26 04:05:57,027 - INFO - query-engine:active 2026-01-26 04:05:57,027 - INFO - webui 2026-01-26 04:05:57,027 - INFO - webui:active 2026-01-26 04:05:57,027 - INFO - webui-middleware:active 2026-01-26 04:05:57,027 - INFO - redis:active 2026-01-26 04:05:57,027 - INFO - analytics_snmp 2026-01-26 04:05:57,035 - INFO - snmp-collector:active 2026-01-26 04:05:57,036 - INFO - snmp-topology:active 2026-01-26 04:05:57,036 - INFO - analytics_alarm 2026-01-26 04:05:57,036 - INFO - alarmgen:active 2026-01-26 04:05:57,180 - INFO - 10.0.0.65 2026-01-26 04:05:57,180 - INFO - vrouter 2026-01-26 04:05:57,186 - INFO - vrouter-nodemgr:active 2026-01-26 04:05:57,193 - INFO - agent:active 2026-01-26 04:05:57,281 - INFO - 10.0.0.130 2026-01-26 04:05:57,281 - INFO - vrouter 2026-01-26 04:05:57,286 - INFO - vrouter-nodemgr:active 2026-01-26 04:05:57,290 - INFO - agent:active 2026-01-26 04:05:57,291 - INFO - All the contrail services are up on all nodes 2026-01-26 04:05:57,291 - INFO - contrail-status passed 2026-01-26 04:05:58,560 - DEBUG - Skipping xmpp flap check
2026-01-26 04:05:58,560 - INFO - END TEST : test_contrail_status : PASSED[0:00:03]
2026-01-26 04:05:58,560 - INFO - --------------------------------------------------------------------------------