2026-04-20 16:42:49,270 - INFO - Using existing project ['default-domain', 'admin'](f007f454-f656-4cc4-8f8f-d27141840441) 2026-04-20 16:42:49,298 - INFO - ================================================================================ 2026-04-20 16:42:49,299 - INFO - STARTING TEST : test_contrail_status 2026-04-20 16:42:49,300 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active 2026-04-20 16:42:49,982 - DEBUG - Nothing to compare xmpp stats {'10.0.0.50': {'10.20.0.193': '0', '10.20.0.25': '0'}, '10.0.0.130': {'10.20.0.193': '0', '10.20.0.25': '0'}} with 2026-04-20 16:42:49,982 - INFO - Initial checks done. Running the testcase now 2026-04-20 16:42:49,982 - INFO - 2026-04-20 16:42:50,065 - INFO - 10.0.0.23 2026-04-20 16:42:50,065 - INFO - config 2026-04-20 16:42:50,073 - INFO - config-nodemgr:active 2026-04-20 16:42:50,079 - INFO - api-server:active 2026-04-20 16:42:50,085 - INFO - schema:backup 2026-04-20 16:42:50,089 - INFO - svc-monitor:backup 2026-04-20 16:42:50,094 - INFO - device-manager:backup 2026-04-20 16:42:50,094 - INFO - config-database 2026-04-20 16:42:50,094 - INFO - config-cassandra:active 2026-04-20 16:42:50,094 - INFO - config-zookeeper:active 2026-04-20 16:42:50,094 - INFO - config-rabbitmq:active 2026-04-20 16:42:50,094 - INFO - control 2026-04-20 16:42:50,101 - INFO - control-nodemgr:active 2026-04-20 16:42:50,108 - INFO - control:active 2026-04-20 16:42:50,108 - INFO - named:active 2026-04-20 16:42:50,114 - INFO - dns:active 2026-04-20 16:42:50,114 - INFO - analytics 2026-04-20 16:42:50,120 - INFO - analytics-nodemgr:active 2026-04-20 16:42:50,125 - INFO - analytics-api:active 2026-04-20 16:42:50,131 - INFO - collector:active 2026-04-20 16:42:50,131 - INFO - analytics-database 2026-04-20 16:42:50,131 - INFO - analytics-cassandra:active 2026-04-20 16:42:50,138 - INFO - analyticsdb-nodemgr:active 2026-04-20 16:42:50,144 - INFO - query-engine:active 2026-04-20 16:42:50,144 - INFO - webui 2026-04-20 16:42:50,144 - INFO - webui:active 2026-04-20 16:42:50,144 - INFO - webui-middleware:active 2026-04-20 16:42:50,144 - INFO - redis:active 2026-04-20 16:42:50,144 - INFO - analytics_snmp 2026-04-20 16:42:50,149 - INFO - snmp-collector:active 2026-04-20 16:42:50,149 - INFO - snmp-topology:active 2026-04-20 16:42:50,150 - INFO - analytics_alarm 2026-04-20 16:42:50,150 - INFO - alarmgen:active 2026-04-20 16:42:50,207 - INFO - 10.0.0.249 2026-04-20 16:42:50,207 - INFO - config 2026-04-20 16:42:50,214 - INFO - config-nodemgr:active 2026-04-20 16:42:50,221 - INFO - api-server:active 2026-04-20 16:42:50,227 - INFO - schema:backup 2026-04-20 16:42:50,232 - INFO - svc-monitor:backup 2026-04-20 16:42:50,238 - INFO - device-manager:active 2026-04-20 16:42:50,238 - INFO - config-database 2026-04-20 16:42:50,238 - INFO - config-cassandra:active 2026-04-20 16:42:50,238 - INFO - config-zookeeper:active 2026-04-20 16:42:50,238 - INFO - config-rabbitmq:active 2026-04-20 16:42:50,238 - INFO - control 2026-04-20 16:42:50,244 - INFO - control-nodemgr:active 2026-04-20 16:42:50,251 - INFO - control:active 2026-04-20 16:42:50,251 - INFO - named:active 2026-04-20 16:42:50,257 - INFO - dns:active 2026-04-20 16:42:50,257 - INFO - analytics 2026-04-20 16:42:50,263 - INFO - analytics-nodemgr:active 2026-04-20 16:42:50,268 - INFO - analytics-api:active 2026-04-20 16:42:50,274 - INFO - collector:active 2026-04-20 16:42:50,274 - INFO - analytics-database 2026-04-20 16:42:50,274 - INFO - analytics-cassandra:active 2026-04-20 16:42:50,281 - INFO - analyticsdb-nodemgr:active 2026-04-20 16:42:50,288 - INFO - query-engine:active 2026-04-20 16:42:50,288 - INFO - webui 2026-04-20 16:42:50,288 - INFO - webui:active 2026-04-20 16:42:50,288 - INFO - webui-middleware:active 2026-04-20 16:42:50,288 - INFO - redis:active 2026-04-20 16:42:50,289 - INFO - analytics_snmp 2026-04-20 16:42:50,294 - INFO - snmp-collector:active 2026-04-20 16:42:50,295 - INFO - snmp-topology:active 2026-04-20 16:42:50,295 - INFO - analytics_alarm 2026-04-20 16:42:50,295 - INFO - alarmgen:active 2026-04-20 16:42:50,352 - INFO - 10.0.0.65 2026-04-20 16:42:50,353 - INFO - config 2026-04-20 16:42:50,361 - INFO - config-nodemgr:active 2026-04-20 16:42:50,367 - INFO - api-server:active 2026-04-20 16:42:50,372 - INFO - schema:active 2026-04-20 16:42:50,380 - INFO - svc-monitor:active 2026-04-20 16:42:50,385 - INFO - device-manager:backup 2026-04-20 16:42:50,385 - INFO - config-database 2026-04-20 16:42:50,385 - INFO - config-cassandra:active 2026-04-20 16:42:50,385 - INFO - config-zookeeper:active 2026-04-20 16:42:50,385 - INFO - config-rabbitmq:active 2026-04-20 16:42:50,385 - INFO - control 2026-04-20 16:42:50,392 - INFO - control-nodemgr:active 2026-04-20 16:42:50,398 - INFO - control:active 2026-04-20 16:42:50,398 - INFO - named:active 2026-04-20 16:42:50,404 - INFO - dns:active 2026-04-20 16:42:50,404 - INFO - analytics 2026-04-20 16:42:50,415 - INFO - analytics-nodemgr:active 2026-04-20 16:42:50,423 - INFO - analytics-api:active 2026-04-20 16:42:50,433 - INFO - collector:active 2026-04-20 16:42:50,433 - INFO - analytics-database 2026-04-20 16:42:50,433 - INFO - analytics-cassandra:active 2026-04-20 16:42:50,441 - INFO - analyticsdb-nodemgr:active 2026-04-20 16:42:50,449 - INFO - query-engine:active 2026-04-20 16:42:50,449 - INFO - webui 2026-04-20 16:42:50,449 - INFO - webui:active 2026-04-20 16:42:50,449 - INFO - webui-middleware:active 2026-04-20 16:42:50,449 - INFO - redis:active 2026-04-20 16:42:50,449 - INFO - analytics_snmp 2026-04-20 16:42:50,457 - INFO - snmp-collector:active 2026-04-20 16:42:50,457 - INFO - snmp-topology:active 2026-04-20 16:42:50,457 - INFO - analytics_alarm 2026-04-20 16:42:50,457 - INFO - alarmgen:active 2026-04-20 16:42:50,516 - INFO - 10.0.0.50 2026-04-20 16:42:50,516 - INFO - vrouter 2026-04-20 16:42:50,523 - INFO - vrouter-nodemgr:active 2026-04-20 16:42:50,530 - INFO - agent:active 2026-04-20 16:42:50,577 - INFO - 10.0.0.130 2026-04-20 16:42:50,577 - INFO - vrouter 2026-04-20 16:42:50,583 - INFO - vrouter-nodemgr:active 2026-04-20 16:42:50,590 - INFO - agent:active 2026-04-20 16:42:50,590 - INFO - All the contrail services are up on all nodes 2026-04-20 16:42:50,590 - INFO - contrail-status passed 2026-04-20 16:42:51,242 - DEBUG - No XMPP flaps were noticed during the test 2026-04-20 16:42:51,242 - INFO - END TEST : test_contrail_status : PASSED[0:00:02] 2026-04-20 16:42:51,242 - INFO - -------------------------------------------------------------------------------- 2026-04-20 18:51:10,397 - INFO - Using existing project ['default-domain', 'admin'](f007f454-f656-4cc4-8f8f-d27141840441) 2026-04-20 18:51:10,408 - INFO - ================================================================================ 2026-04-20 18:51:10,408 - INFO - STARTING TEST : test_contrail_status 2026-04-20 18:51:10,408 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active 2026-04-20 18:51:11,681 - DEBUG - Skipping xmpp flap check 2026-04-20 18:51:11,682 - INFO - Initial checks done. Running the testcase now 2026-04-20 18:51:11,682 - INFO - 2026-04-20 18:51:11,996 - INFO - 10.0.0.23 2026-04-20 18:51:11,996 - INFO - config 2026-04-20 18:51:12,007 - INFO - config-nodemgr:active 2026-04-20 18:51:12,014 - INFO - api-server:active 2026-04-20 18:51:12,019 - INFO - schema:backup 2026-04-20 18:51:12,025 - INFO - svc-monitor:backup 2026-04-20 18:51:12,029 - INFO - device-manager:backup 2026-04-20 18:51:12,029 - INFO - config-database 2026-04-20 18:51:12,029 - INFO - config-cassandra:active 2026-04-20 18:51:12,029 - INFO - config-zookeeper:active 2026-04-20 18:51:12,029 - INFO - config-rabbitmq:active 2026-04-20 18:51:12,030 - INFO - control 2026-04-20 18:51:12,038 - INFO - control-nodemgr:active 2026-04-20 18:51:12,045 - INFO - control:active 2026-04-20 18:51:12,045 - INFO - named:active 2026-04-20 18:51:12,062 - INFO - dns:active 2026-04-20 18:51:12,062 - INFO - analytics 2026-04-20 18:51:12,068 - INFO - analytics-nodemgr:active 2026-04-20 18:51:12,075 - INFO - analytics-api:active 2026-04-20 18:51:12,097 - INFO - collector:active 2026-04-20 18:51:12,098 - INFO - analytics-database 2026-04-20 18:51:12,098 - INFO - analytics-cassandra:active 2026-04-20 18:51:12,104 - INFO - analyticsdb-nodemgr:active 2026-04-20 18:51:12,134 - INFO - query-engine:active 2026-04-20 18:51:12,135 - INFO - webui 2026-04-20 18:51:12,135 - INFO - webui:active 2026-04-20 18:51:12,135 - INFO - webui-middleware:active 2026-04-20 18:51:12,135 - INFO - redis:active 2026-04-20 18:51:12,136 - INFO - analytics_snmp 2026-04-20 18:51:12,141 - INFO - snmp-collector:active 2026-04-20 18:51:12,141 - INFO - snmp-topology:active 2026-04-20 18:51:12,141 - INFO - analytics_alarm 2026-04-20 18:51:12,141 - INFO - alarmgen:active 2026-04-20 18:51:12,273 - INFO - 10.0.0.249 2026-04-20 18:51:12,274 - INFO - config 2026-04-20 18:51:12,281 - INFO - config-nodemgr:active 2026-04-20 18:51:12,286 - INFO - api-server:active 2026-04-20 18:51:12,300 - INFO - schema:backup 2026-04-20 18:51:12,312 - INFO - svc-monitor:backup 2026-04-20 18:51:12,318 - INFO - device-manager:active 2026-04-20 18:51:12,319 - INFO - config-database 2026-04-20 18:51:12,319 - INFO - config-cassandra:active 2026-04-20 18:51:12,319 - INFO - config-zookeeper:active 2026-04-20 18:51:12,319 - INFO - config-rabbitmq:active 2026-04-20 18:51:12,319 - INFO - control 2026-04-20 18:51:12,326 - INFO - control-nodemgr:active 2026-04-20 18:51:12,334 - INFO - control:active 2026-04-20 18:51:12,334 - INFO - named:active 2026-04-20 18:51:12,340 - INFO - dns:active 2026-04-20 18:51:12,340 - INFO - analytics 2026-04-20 18:51:12,346 - INFO - analytics-nodemgr:active 2026-04-20 18:51:12,351 - INFO - analytics-api:active 2026-04-20 18:51:12,358 - INFO - collector:active 2026-04-20 18:51:12,359 - INFO - analytics-database 2026-04-20 18:51:12,359 - INFO - analytics-cassandra:active 2026-04-20 18:51:12,366 - INFO - analyticsdb-nodemgr:active 2026-04-20 18:51:12,374 - INFO - query-engine:active 2026-04-20 18:51:12,375 - INFO - webui 2026-04-20 18:51:12,375 - INFO - webui:active 2026-04-20 18:51:12,375 - INFO - webui-middleware:active 2026-04-20 18:51:12,376 - INFO - redis:active 2026-04-20 18:51:12,376 - INFO - analytics_snmp 2026-04-20 18:51:12,385 - INFO - snmp-collector:active 2026-04-20 18:51:12,385 - INFO - snmp-topology:active 2026-04-20 18:51:12,385 - INFO - analytics_alarm 2026-04-20 18:51:12,386 - INFO - alarmgen:active 2026-04-20 18:51:12,512 - INFO - 10.0.0.65 2026-04-20 18:51:12,512 - INFO - config 2026-04-20 18:51:12,519 - INFO - config-nodemgr:active 2026-04-20 18:51:12,551 - INFO - api-server:active 2026-04-20 18:51:12,565 - INFO - schema:active 2026-04-20 18:51:12,572 - INFO - svc-monitor:active 2026-04-20 18:51:12,579 - INFO - device-manager:backup 2026-04-20 18:51:12,580 - INFO - config-database 2026-04-20 18:51:12,580 - INFO - config-cassandra:active 2026-04-20 18:51:12,580 - INFO - config-zookeeper:active 2026-04-20 18:51:12,580 - INFO - config-rabbitmq:active 2026-04-20 18:51:12,581 - INFO - control 2026-04-20 18:51:12,597 - INFO - control-nodemgr:active 2026-04-20 18:51:12,612 - INFO - control:active 2026-04-20 18:51:12,614 - INFO - named:active 2026-04-20 18:51:12,624 - INFO - dns:active 2026-04-20 18:51:12,625 - INFO - analytics 2026-04-20 18:51:12,635 - INFO - analytics-nodemgr:active 2026-04-20 18:51:12,650 - INFO - analytics-api:active 2026-04-20 18:51:12,662 - INFO - collector:active 2026-04-20 18:51:12,664 - INFO - analytics-database 2026-04-20 18:51:12,664 - INFO - analytics-cassandra:active 2026-04-20 18:51:12,679 - INFO - analyticsdb-nodemgr:active 2026-04-20 18:51:12,693 - INFO - query-engine:active 2026-04-20 18:51:12,694 - INFO - webui 2026-04-20 18:51:12,694 - INFO - webui:active 2026-04-20 18:51:12,694 - INFO - webui-middleware:active 2026-04-20 18:51:12,694 - INFO - redis:active 2026-04-20 18:51:12,694 - INFO - analytics_snmp 2026-04-20 18:51:12,699 - INFO - snmp-collector:initializing (Database:RabbitMQ[] connection down) 2026-04-20 18:51:12,700 - INFO - snmp-topology:active 2026-04-20 18:51:12,700 - INFO - analytics_alarm 2026-04-20 18:51:12,700 - INFO - alarmgen:active 2026-04-20 18:51:12,799 - INFO - 10.0.0.50 2026-04-20 18:51:12,799 - INFO - vrouter 2026-04-20 18:51:12,805 - INFO - vrouter-nodemgr:active 2026-04-20 18:51:12,814 - INFO - agent:active 2026-04-20 18:51:12,916 - INFO - 10.0.0.130 2026-04-20 18:51:12,916 - INFO - vrouter 2026-04-20 18:51:12,924 - INFO - vrouter-nodemgr:active 2026-04-20 18:51:12,932 - INFO - agent:active 2026-04-20 18:51:12,932 - DEBUG - defaultdict(, {'10.0.0.65': {'snmp-collector': {'status': 'initializing', 'description': 'Database:RabbitMQ[] connection down'}}}) 2026-04-20 18:51:12,932 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 0 2026-04-20 18:51:23,062 - INFO - 10.0.0.23 2026-04-20 18:51:23,062 - INFO - config 2026-04-20 18:51:23,070 - INFO - config-nodemgr:active 2026-04-20 18:51:23,080 - INFO - api-server:active 2026-04-20 18:51:23,088 - INFO - schema:backup 2026-04-20 18:51:23,092 - INFO - svc-monitor:backup 2026-04-20 18:51:23,096 - INFO - device-manager:backup 2026-04-20 18:51:23,096 - INFO - config-database 2026-04-20 18:51:23,097 - INFO - config-cassandra:active 2026-04-20 18:51:23,097 - INFO - config-zookeeper:active 2026-04-20 18:51:23,097 - INFO - config-rabbitmq:active 2026-04-20 18:51:23,097 - INFO - control 2026-04-20 18:51:23,103 - INFO - control-nodemgr:active 2026-04-20 18:51:23,110 - INFO - control:active 2026-04-20 18:51:23,111 - INFO - named:active 2026-04-20 18:51:23,118 - INFO - dns:active 2026-04-20 18:51:23,118 - INFO - analytics 2026-04-20 18:51:23,125 - INFO - analytics-nodemgr:active 2026-04-20 18:51:23,131 - INFO - analytics-api:active 2026-04-20 18:51:23,137 - INFO - collector:active 2026-04-20 18:51:23,137 - INFO - analytics-database 2026-04-20 18:51:23,137 - INFO - analytics-cassandra:active 2026-04-20 18:51:23,145 - INFO - analyticsdb-nodemgr:active 2026-04-20 18:51:23,153 - INFO - query-engine:active 2026-04-20 18:51:23,153 - INFO - webui 2026-04-20 18:51:23,153 - INFO - webui:active 2026-04-20 18:51:23,153 - INFO - webui-middleware:active 2026-04-20 18:51:23,154 - INFO - redis:active 2026-04-20 18:51:23,154 - INFO - analytics_snmp 2026-04-20 18:51:23,159 - INFO - snmp-collector:active 2026-04-20 18:51:23,159 - INFO - snmp-topology:active 2026-04-20 18:51:23,160 - INFO - analytics_alarm 2026-04-20 18:51:23,160 - INFO - alarmgen:active 2026-04-20 18:51:23,283 - INFO - 10.0.0.249 2026-04-20 18:51:23,283 - INFO - config 2026-04-20 18:51:23,291 - INFO - config-nodemgr:active 2026-04-20 18:51:23,298 - INFO - api-server:active 2026-04-20 18:51:23,303 - INFO - schema:backup 2026-04-20 18:51:23,309 - INFO - svc-monitor:backup 2026-04-20 18:51:23,315 - INFO - device-manager:active 2026-04-20 18:51:23,316 - INFO - config-database 2026-04-20 18:51:23,316 - INFO - config-cassandra:active 2026-04-20 18:51:23,316 - INFO - config-zookeeper:active 2026-04-20 18:51:23,316 - INFO - config-rabbitmq:active 2026-04-20 18:51:23,316 - INFO - control 2026-04-20 18:51:23,323 - INFO - control-nodemgr:active 2026-04-20 18:51:23,338 - INFO - control:active 2026-04-20 18:51:23,338 - INFO - named:active 2026-04-20 18:51:23,349 - INFO - dns:active 2026-04-20 18:51:23,349 - INFO - analytics 2026-04-20 18:51:23,358 - INFO - analytics-nodemgr:active 2026-04-20 18:51:23,364 - INFO - analytics-api:active 2026-04-20 18:51:23,371 - INFO - collector:active 2026-04-20 18:51:23,371 - INFO - analytics-database 2026-04-20 18:51:23,371 - INFO - analytics-cassandra:active 2026-04-20 18:51:23,378 - INFO - analyticsdb-nodemgr:active 2026-04-20 18:51:23,386 - INFO - query-engine:active 2026-04-20 18:51:23,386 - INFO - webui 2026-04-20 18:51:23,387 - INFO - webui:active 2026-04-20 18:51:23,387 - INFO - webui-middleware:active 2026-04-20 18:51:23,387 - INFO - redis:active 2026-04-20 18:51:23,387 - INFO - analytics_snmp 2026-04-20 18:51:23,395 - INFO - snmp-collector:active 2026-04-20 18:51:23,395 - INFO - snmp-topology:active 2026-04-20 18:51:23,395 - INFO - analytics_alarm 2026-04-20 18:51:23,395 - INFO - alarmgen:active 2026-04-20 18:51:23,559 - INFO - 10.0.0.65 2026-04-20 18:51:23,559 - INFO - config 2026-04-20 18:51:23,565 - INFO - config-nodemgr:active 2026-04-20 18:51:23,571 - INFO - api-server:active 2026-04-20 18:51:23,576 - INFO - schema:active 2026-04-20 18:51:23,582 - INFO - svc-monitor:active 2026-04-20 18:51:23,588 - INFO - device-manager:backup 2026-04-20 18:51:23,588 - INFO - config-database 2026-04-20 18:51:23,589 - INFO - config-cassandra:active 2026-04-20 18:51:23,589 - INFO - config-zookeeper:active 2026-04-20 18:51:23,589 - INFO - config-rabbitmq:active 2026-04-20 18:51:23,589 - INFO - control 2026-04-20 18:51:23,596 - INFO - control-nodemgr:active 2026-04-20 18:51:23,606 - INFO - control:active 2026-04-20 18:51:23,607 - INFO - named:active 2026-04-20 18:51:23,613 - INFO - dns:active 2026-04-20 18:51:23,614 - INFO - analytics 2026-04-20 18:51:23,620 - INFO - analytics-nodemgr:active 2026-04-20 18:51:23,626 - INFO - analytics-api:active 2026-04-20 18:51:23,634 - INFO - collector:active 2026-04-20 18:51:23,634 - INFO - analytics-database 2026-04-20 18:51:23,635 - INFO - analytics-cassandra:active 2026-04-20 18:51:23,651 - INFO - analyticsdb-nodemgr:active 2026-04-20 18:51:23,659 - INFO - query-engine:active 2026-04-20 18:51:23,659 - INFO - webui 2026-04-20 18:51:23,659 - INFO - webui:active 2026-04-20 18:51:23,659 - INFO - webui-middleware:active 2026-04-20 18:51:23,659 - INFO - redis:active 2026-04-20 18:51:23,659 - INFO - analytics_snmp 2026-04-20 18:51:23,665 - INFO - snmp-collector:active 2026-04-20 18:51:23,665 - INFO - snmp-topology:active 2026-04-20 18:51:23,665 - INFO - analytics_alarm 2026-04-20 18:51:23,665 - INFO - alarmgen:active 2026-04-20 18:51:23,760 - INFO - 10.0.0.50 2026-04-20 18:51:23,760 - INFO - vrouter 2026-04-20 18:51:23,768 - INFO - vrouter-nodemgr:active 2026-04-20 18:51:23,775 - INFO - agent:active 2026-04-20 18:51:23,865 - INFO - 10.0.0.130 2026-04-20 18:51:23,865 - INFO - vrouter 2026-04-20 18:51:23,870 - INFO - vrouter-nodemgr:active 2026-04-20 18:51:23,876 - INFO - agent:active 2026-04-20 18:51:23,877 - INFO - All the contrail services are up on all nodes 2026-04-20 18:51:23,877 - INFO - contrail-status passed 2026-04-20 18:51:25,197 - DEBUG - Skipping xmpp flap check 2026-04-20 18:51:25,198 - INFO - END TEST : test_contrail_status : PASSED[0:00:15] 2026-04-20 18:51:25,198 - INFO - --------------------------------------------------------------------------------