2026-04-06 02:29:02,801 - INFO - Using existing project ['default-domain', 'admin'](c7eef1ed-1186-4b89-b3a0-92a64280c0ed) 2026-04-06 02:29:02,808 - INFO - ================================================================================ 2026-04-06 02:29:02,809 - INFO - STARTING TEST : test_contrail_status 2026-04-06 02:29:02,809 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active 2026-04-06 02:29:03,448 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.193': '0', '10.20.0.129': '0'}, '10.0.0.241': {'10.20.0.18': '0', '10.20.0.129': '0'}} with 2026-04-06 02:29:03,449 - INFO - Initial checks done. Running the testcase now 2026-04-06 02:29:03,449 - INFO - 2026-04-06 02:29:03,517 - INFO - 10.0.0.249 2026-04-06 02:29:03,517 - INFO - config 2026-04-06 02:29:03,523 - INFO - config-nodemgr:active 2026-04-06 02:29:03,527 - INFO - api-server:active 2026-04-06 02:29:03,530 - INFO - schema:backup 2026-04-06 02:29:03,533 - INFO - svc-monitor:active 2026-04-06 02:29:03,537 - INFO - device-manager:active 2026-04-06 02:29:03,537 - INFO - config-database 2026-04-06 02:29:03,537 - INFO - config-cassandra:active 2026-04-06 02:29:03,537 - INFO - config-zookeeper:active 2026-04-06 02:29:03,537 - INFO - config-rabbitmq:active 2026-04-06 02:29:03,537 - INFO - control 2026-04-06 02:29:03,542 - INFO - control-nodemgr:active 2026-04-06 02:29:03,545 - INFO - control:active 2026-04-06 02:29:03,546 - INFO - named:active 2026-04-06 02:29:03,549 - INFO - dns:active 2026-04-06 02:29:03,549 - INFO - analytics 2026-04-06 02:29:03,554 - INFO - analytics-nodemgr:active 2026-04-06 02:29:03,558 - INFO - analytics-api:active 2026-04-06 02:29:03,563 - INFO - collector:active 2026-04-06 02:29:03,563 - INFO - analytics-database 2026-04-06 02:29:03,563 - INFO - analytics-cassandra:active 2026-04-06 02:29:03,567 - INFO - analyticsdb-nodemgr:active 2026-04-06 02:29:03,572 - INFO - query-engine:active 2026-04-06 02:29:03,572 - INFO - webui 2026-04-06 02:29:03,572 - INFO - webui:active 2026-04-06 02:29:03,572 - INFO - webui-middleware:active 2026-04-06 02:29:03,572 - INFO - redis:active 2026-04-06 02:29:03,572 - INFO - analytics_snmp 2026-04-06 02:29:03,576 - INFO - snmp-collector:active 2026-04-06 02:29:03,576 - INFO - snmp-topology:active 2026-04-06 02:29:03,577 - INFO - analytics_alarm 2026-04-06 02:29:03,577 - INFO - alarmgen:active 2026-04-06 02:29:03,644 - INFO - 10.0.0.27 2026-04-06 02:29:03,644 - INFO - config 2026-04-06 02:29:03,653 - INFO - config-nodemgr:active 2026-04-06 02:29:03,658 - INFO - api-server:active 2026-04-06 02:29:03,664 - INFO - schema:active 2026-04-06 02:29:03,668 - INFO - svc-monitor:backup 2026-04-06 02:29:03,671 - INFO - device-manager:backup 2026-04-06 02:29:03,672 - INFO - config-database 2026-04-06 02:29:03,672 - INFO - config-cassandra:active 2026-04-06 02:29:03,672 - INFO - config-zookeeper:active 2026-04-06 02:29:03,672 - INFO - config-rabbitmq:active 2026-04-06 02:29:03,672 - INFO - control 2026-04-06 02:29:03,678 - INFO - control-nodemgr:active 2026-04-06 02:30:03,684 - INFO - control:timeout 2026-04-06 02:30:03,684 - INFO - named:active 2026-04-06 02:30:03,690 - INFO - dns:active 2026-04-06 02:30:03,690 - INFO - analytics 2026-04-06 02:30:03,696 - INFO - analytics-nodemgr:active 2026-04-06 02:30:03,702 - INFO - analytics-api:active 2026-04-06 02:30:03,707 - INFO - collector:active 2026-04-06 02:30:03,707 - INFO - analytics-database 2026-04-06 02:30:03,708 - INFO - analytics-cassandra:active 2026-04-06 02:30:03,714 - INFO - analyticsdb-nodemgr:active 2026-04-06 02:30:03,719 - INFO - query-engine:active 2026-04-06 02:30:03,719 - INFO - webui 2026-04-06 02:30:03,719 - INFO - webui:active 2026-04-06 02:30:03,719 - INFO - webui-middleware:active 2026-04-06 02:30:03,719 - INFO - redis:active 2026-04-06 02:30:03,719 - INFO - analytics_snmp 2026-04-06 02:30:03,724 - INFO - snmp-collector:active 2026-04-06 02:30:03,724 - INFO - snmp-topology:active 2026-04-06 02:30:03,724 - INFO - analytics_alarm 2026-04-06 02:30:03,724 - INFO - alarmgen:active 2026-04-06 02:30:03,790 - INFO - 10.0.0.33 2026-04-06 02:30:03,790 - INFO - config 2026-04-06 02:30:03,797 - INFO - config-nodemgr:active 2026-04-06 02:30:03,801 - INFO - api-server:active 2026-04-06 02:30:03,805 - INFO - schema:backup 2026-04-06 02:30:03,808 - INFO - svc-monitor:backup 2026-04-06 02:30:03,811 - INFO - device-manager:backup 2026-04-06 02:30:03,811 - INFO - config-database 2026-04-06 02:30:03,811 - INFO - config-cassandra:active 2026-04-06 02:30:03,811 - INFO - config-zookeeper:active 2026-04-06 02:30:03,812 - INFO - config-rabbitmq:active 2026-04-06 02:30:03,812 - INFO - control 2026-04-06 02:30:03,818 - INFO - control-nodemgr:active 2026-04-06 02:30:03,826 - INFO - control:active 2026-04-06 02:30:03,826 - INFO - named:active 2026-04-06 02:30:03,832 - INFO - dns:active 2026-04-06 02:30:03,832 - INFO - analytics 2026-04-06 02:30:03,839 - INFO - analytics-nodemgr:active 2026-04-06 02:30:03,843 - INFO - analytics-api:active 2026-04-06 02:30:03,852 - INFO - collector:active 2026-04-06 02:30:03,852 - INFO - analytics-database 2026-04-06 02:30:03,852 - INFO - analytics-cassandra:active 2026-04-06 02:30:03,859 - INFO - analyticsdb-nodemgr:active 2026-04-06 02:30:03,865 - INFO - query-engine:active 2026-04-06 02:30:03,866 - INFO - webui 2026-04-06 02:30:03,866 - INFO - webui:active 2026-04-06 02:30:03,866 - INFO - webui-middleware:active 2026-04-06 02:30:03,866 - INFO - redis:active 2026-04-06 02:30:03,866 - INFO - analytics_snmp 2026-04-06 02:30:03,870 - INFO - snmp-collector:active 2026-04-06 02:30:03,871 - INFO - snmp-topology:active 2026-04-06 02:30:03,871 - INFO - analytics_alarm 2026-04-06 02:30:03,871 - INFO - alarmgen:active 2026-04-06 02:30:03,927 - INFO - 10.0.0.19 2026-04-06 02:30:03,927 - INFO - vrouter 2026-04-06 02:30:03,933 - INFO - vrouter-nodemgr:active 2026-04-06 02:30:03,938 - INFO - agent:active 2026-04-06 02:30:03,994 - INFO - 10.0.0.241 2026-04-06 02:30:03,994 - INFO - vrouter 2026-04-06 02:30:04,000 - INFO - vrouter-nodemgr:active 2026-04-06 02:30:04,006 - INFO - agent:active 2026-04-06 02:30:04,006 - DEBUG - defaultdict(, {'10.0.0.27': {'control': {'status': 'timeout', 'description': None}}}) 2026-04-06 02:30:04,006 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 0 2026-04-06 02:30:14,086 - INFO - 10.0.0.249 2026-04-06 02:30:14,086 - INFO - config 2026-04-06 02:30:14,091 - INFO - config-nodemgr:active 2026-04-06 02:30:14,095 - INFO - api-server:active 2026-04-06 02:30:14,097 - INFO - schema:backup 2026-04-06 02:30:14,101 - INFO - svc-monitor:active 2026-04-06 02:30:14,105 - INFO - device-manager:active 2026-04-06 02:30:14,105 - INFO - config-database 2026-04-06 02:30:14,105 - INFO - config-cassandra:active 2026-04-06 02:30:14,105 - INFO - config-zookeeper:active 2026-04-06 02:30:14,105 - INFO - config-rabbitmq:active 2026-04-06 02:30:14,106 - INFO - control 2026-04-06 02:30:14,111 - INFO - control-nodemgr:active 2026-04-06 02:30:14,116 - INFO - control:active 2026-04-06 02:30:14,116 - INFO - named:active 2026-04-06 02:30:14,120 - INFO - dns:active 2026-04-06 02:30:14,120 - INFO - analytics 2026-04-06 02:30:14,127 - INFO - analytics-nodemgr:active 2026-04-06 02:30:14,132 - INFO - analytics-api:active 2026-04-06 02:30:14,137 - INFO - collector:active 2026-04-06 02:30:14,138 - INFO - analytics-database 2026-04-06 02:30:14,138 - INFO - analytics-cassandra:active 2026-04-06 02:30:14,142 - INFO - analyticsdb-nodemgr:active 2026-04-06 02:30:14,148 - INFO - query-engine:active 2026-04-06 02:30:14,148 - INFO - webui 2026-04-06 02:30:14,148 - INFO - webui:active 2026-04-06 02:30:14,148 - INFO - webui-middleware:active 2026-04-06 02:30:14,148 - INFO - redis:active 2026-04-06 02:30:14,148 - INFO - analytics_snmp 2026-04-06 02:30:14,153 - INFO - snmp-collector:active 2026-04-06 02:30:14,153 - INFO - snmp-topology:active 2026-04-06 02:30:14,153 - INFO - analytics_alarm 2026-04-06 02:30:14,153 - INFO - alarmgen:active 2026-04-06 02:30:14,220 - INFO - 10.0.0.27 2026-04-06 02:30:14,220 - INFO - config 2026-04-06 02:30:14,226 - INFO - config-nodemgr:active 2026-04-06 02:30:14,231 - INFO - api-server:active 2026-04-06 02:30:14,235 - INFO - schema:active 2026-04-06 02:30:14,238 - INFO - svc-monitor:backup 2026-04-06 02:30:14,243 - INFO - device-manager:backup 2026-04-06 02:30:14,244 - INFO - config-database 2026-04-06 02:30:14,244 - INFO - config-cassandra:active 2026-04-06 02:30:14,245 - INFO - config-zookeeper:active 2026-04-06 02:30:14,245 - INFO - config-rabbitmq:active 2026-04-06 02:30:14,245 - INFO - control 2026-04-06 02:30:14,252 - INFO - control-nodemgr:active 2026-04-06 02:30:14,258 - INFO - control:active 2026-04-06 02:30:14,259 - INFO - named:active 2026-04-06 02:30:14,263 - INFO - dns:active 2026-04-06 02:30:14,264 - INFO - analytics 2026-04-06 02:30:14,271 - INFO - analytics-nodemgr:active 2026-04-06 02:30:14,276 - INFO - analytics-api:active 2026-04-06 02:30:14,281 - INFO - collector:active 2026-04-06 02:30:14,281 - INFO - analytics-database 2026-04-06 02:30:14,282 - INFO - analytics-cassandra:active 2026-04-06 02:30:14,287 - INFO - analyticsdb-nodemgr:active 2026-04-06 02:30:14,294 - INFO - query-engine:active 2026-04-06 02:30:14,295 - INFO - webui 2026-04-06 02:30:14,295 - INFO - webui:active 2026-04-06 02:30:14,295 - INFO - webui-middleware:active 2026-04-06 02:30:14,295 - INFO - redis:active 2026-04-06 02:30:14,295 - INFO - analytics_snmp 2026-04-06 02:30:14,301 - INFO - snmp-collector:active 2026-04-06 02:30:14,301 - INFO - snmp-topology:active 2026-04-06 02:30:14,302 - INFO - analytics_alarm 2026-04-06 02:30:14,302 - INFO - alarmgen:active 2026-04-06 02:30:14,370 - INFO - 10.0.0.33 2026-04-06 02:30:14,370 - INFO - config 2026-04-06 02:30:14,378 - INFO - config-nodemgr:active 2026-04-06 02:30:14,384 - INFO - api-server:active 2026-04-06 02:30:14,388 - INFO - schema:backup 2026-04-06 02:30:14,392 - INFO - svc-monitor:backup 2026-04-06 02:30:14,396 - INFO - device-manager:backup 2026-04-06 02:30:14,396 - INFO - config-database 2026-04-06 02:30:14,396 - INFO - config-cassandra:active 2026-04-06 02:30:14,396 - INFO - config-zookeeper:active 2026-04-06 02:30:14,396 - INFO - config-rabbitmq:active 2026-04-06 02:30:14,396 - INFO - control 2026-04-06 02:30:14,402 - INFO - control-nodemgr:active 2026-04-06 02:30:14,408 - INFO - control:active 2026-04-06 02:30:14,408 - INFO - named:active 2026-04-06 02:30:14,414 - INFO - dns:active 2026-04-06 02:30:14,414 - INFO - analytics 2026-04-06 02:30:14,420 - INFO - analytics-nodemgr:active 2026-04-06 02:30:14,425 - INFO - analytics-api:active 2026-04-06 02:30:14,431 - INFO - collector:active 2026-04-06 02:30:14,432 - INFO - analytics-database 2026-04-06 02:30:14,432 - INFO - analytics-cassandra:active 2026-04-06 02:30:14,438 - INFO - analyticsdb-nodemgr:active 2026-04-06 02:30:14,447 - INFO - query-engine:active 2026-04-06 02:30:14,447 - INFO - webui 2026-04-06 02:30:14,447 - INFO - webui:active 2026-04-06 02:30:14,447 - INFO - webui-middleware:active 2026-04-06 02:30:14,447 - INFO - redis:active 2026-04-06 02:30:14,447 - INFO - analytics_snmp 2026-04-06 02:30:14,453 - INFO - snmp-collector:active 2026-04-06 02:30:14,453 - INFO - snmp-topology:active 2026-04-06 02:30:14,454 - INFO - analytics_alarm 2026-04-06 02:30:14,454 - INFO - alarmgen:active 2026-04-06 02:30:14,509 - INFO - 10.0.0.19 2026-04-06 02:30:14,510 - INFO - vrouter 2026-04-06 02:30:14,516 - INFO - vrouter-nodemgr:active 2026-04-06 02:30:14,522 - INFO - agent:active 2026-04-06 02:30:14,568 - INFO - 10.0.0.241 2026-04-06 02:30:14,568 - INFO - vrouter 2026-04-06 02:30:14,577 - INFO - vrouter-nodemgr:active 2026-04-06 02:30:14,582 - INFO - agent:active 2026-04-06 02:30:14,582 - INFO - All the contrail services are up on all nodes 2026-04-06 02:30:14,582 - INFO - contrail-status passed 2026-04-06 02:30:15,227 - DEBUG - No XMPP flaps were noticed during the test 2026-04-06 02:30:15,227 - INFO - END TEST : test_contrail_status : PASSED[0:01:13] 2026-04-06 02:30:15,227 - INFO - -------------------------------------------------------------------------------- 2026-04-06 04:31:53,480 - INFO - Using existing project ['default-domain', 'admin'](c7eef1ed-1186-4b89-b3a0-92a64280c0ed) 2026-04-06 04:31:53,492 - INFO - ================================================================================ 2026-04-06 04:31:53,492 - INFO - STARTING TEST : test_contrail_status 2026-04-06 04:31:53,492 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active 2026-04-06 04:31:54,747 - DEBUG - Skipping xmpp flap check 2026-04-06 04:31:54,748 - INFO - Initial checks done. Running the testcase now 2026-04-06 04:31:54,748 - INFO - 2026-04-06 04:31:54,960 - INFO - 10.0.0.249 2026-04-06 04:31:54,960 - INFO - config 2026-04-06 04:31:54,966 - INFO - config-nodemgr:active 2026-04-06 04:31:54,970 - INFO - api-server:active 2026-04-06 04:31:54,973 - INFO - schema:backup 2026-04-06 04:31:54,977 - INFO - svc-monitor:active 2026-04-06 04:31:54,981 - INFO - device-manager:active 2026-04-06 04:31:54,982 - INFO - config-database 2026-04-06 04:31:54,982 - INFO - config-cassandra:active 2026-04-06 04:31:54,982 - INFO - config-zookeeper:active 2026-04-06 04:31:54,982 - INFO - config-rabbitmq:active 2026-04-06 04:31:54,982 - INFO - control 2026-04-06 04:31:54,987 - INFO - control-nodemgr:active 2026-04-06 04:31:54,992 - INFO - control:active 2026-04-06 04:31:54,992 - INFO - named:active 2026-04-06 04:31:55,001 - INFO - dns:active 2026-04-06 04:31:55,001 - INFO - analytics 2026-04-06 04:31:55,005 - INFO - analytics-nodemgr:active 2026-04-06 04:31:55,008 - INFO - analytics-api:active 2026-04-06 04:31:55,025 - INFO - collector:active 2026-04-06 04:31:55,025 - INFO - analytics-database 2026-04-06 04:31:55,026 - INFO - analytics-cassandra:active 2026-04-06 04:31:55,031 - INFO - analyticsdb-nodemgr:active 2026-04-06 04:31:55,059 - INFO - query-engine:active 2026-04-06 04:31:55,059 - INFO - webui 2026-04-06 04:31:55,059 - INFO - webui:active 2026-04-06 04:31:55,060 - INFO - webui-middleware:active 2026-04-06 04:31:55,060 - INFO - redis:active 2026-04-06 04:31:55,060 - INFO - analytics_snmp 2026-04-06 04:31:55,064 - INFO - snmp-collector:active 2026-04-06 04:31:55,064 - INFO - snmp-topology:active 2026-04-06 04:31:55,065 - INFO - analytics_alarm 2026-04-06 04:31:55,065 - INFO - alarmgen:active 2026-04-06 04:31:55,165 - INFO - 10.0.0.27 2026-04-06 04:31:55,165 - INFO - config 2026-04-06 04:31:55,171 - INFO - config-nodemgr:active 2026-04-06 04:31:55,175 - INFO - api-server:active 2026-04-06 04:31:55,180 - INFO - schema:active 2026-04-06 04:31:55,183 - INFO - svc-monitor:backup 2026-04-06 04:31:55,186 - INFO - device-manager:backup 2026-04-06 04:31:55,186 - INFO - config-database 2026-04-06 04:31:55,186 - INFO - config-cassandra:active 2026-04-06 04:31:55,186 - INFO - config-zookeeper:active 2026-04-06 04:31:55,186 - INFO - config-rabbitmq:active 2026-04-06 04:31:55,186 - INFO - control 2026-04-06 04:31:55,191 - INFO - control-nodemgr:active 2026-04-06 04:31:55,196 - INFO - control:active 2026-04-06 04:31:55,196 - INFO - named:active 2026-04-06 04:31:55,200 - INFO - dns:active 2026-04-06 04:31:55,201 - INFO - analytics 2026-04-06 04:31:55,205 - INFO - analytics-nodemgr:active 2026-04-06 04:31:55,209 - INFO - analytics-api:active 2026-04-06 04:31:55,213 - INFO - collector:active 2026-04-06 04:31:55,213 - INFO - analytics-database 2026-04-06 04:31:55,213 - INFO - analytics-cassandra:active 2026-04-06 04:31:55,218 - INFO - analyticsdb-nodemgr:active 2026-04-06 04:31:55,222 - INFO - query-engine:active 2026-04-06 04:31:55,222 - INFO - webui 2026-04-06 04:31:55,222 - INFO - webui:active 2026-04-06 04:31:55,222 - INFO - webui-middleware:active 2026-04-06 04:31:55,222 - INFO - redis:active 2026-04-06 04:31:55,222 - INFO - analytics_snmp 2026-04-06 04:31:55,226 - INFO - snmp-collector:active 2026-04-06 04:31:55,226 - INFO - snmp-topology:active 2026-04-06 04:31:55,226 - INFO - analytics_alarm 2026-04-06 04:31:55,226 - INFO - alarmgen:active 2026-04-06 04:31:55,333 - INFO - 10.0.0.33 2026-04-06 04:31:55,333 - INFO - config 2026-04-06 04:31:55,340 - INFO - config-nodemgr:active 2026-04-06 04:31:55,345 - INFO - api-server:active 2026-04-06 04:31:55,349 - INFO - schema:backup 2026-04-06 04:31:55,352 - INFO - svc-monitor:backup 2026-04-06 04:31:55,356 - INFO - device-manager:backup 2026-04-06 04:31:55,356 - INFO - config-database 2026-04-06 04:31:55,356 - INFO - config-cassandra:active 2026-04-06 04:31:55,356 - INFO - config-zookeeper:active 2026-04-06 04:31:55,357 - INFO - config-rabbitmq:active 2026-04-06 04:31:55,357 - INFO - control 2026-04-06 04:31:55,362 - INFO - control-nodemgr:active 2026-04-06 04:31:55,369 - INFO - control:active 2026-04-06 04:31:55,369 - INFO - named:active 2026-04-06 04:31:55,374 - INFO - dns:active 2026-04-06 04:31:55,374 - INFO - analytics 2026-04-06 04:31:55,379 - INFO - analytics-nodemgr:active 2026-04-06 04:31:55,383 - INFO - analytics-api:active 2026-04-06 04:31:55,388 - INFO - collector:active 2026-04-06 04:31:55,389 - INFO - analytics-database 2026-04-06 04:31:55,389 - INFO - analytics-cassandra:active 2026-04-06 04:31:55,394 - INFO - analyticsdb-nodemgr:active 2026-04-06 04:31:55,399 - INFO - query-engine:active 2026-04-06 04:31:55,399 - INFO - webui 2026-04-06 04:31:55,399 - INFO - webui:active 2026-04-06 04:31:55,400 - INFO - webui-middleware:active 2026-04-06 04:31:55,400 - INFO - redis:active 2026-04-06 04:31:55,400 - INFO - analytics_snmp 2026-04-06 04:31:55,403 - INFO - snmp-collector:active 2026-04-06 04:31:55,404 - INFO - snmp-topology:active 2026-04-06 04:31:55,404 - INFO - analytics_alarm 2026-04-06 04:31:55,404 - INFO - alarmgen:active 2026-04-06 04:31:55,503 - INFO - 10.0.0.19 2026-04-06 04:31:55,503 - INFO - vrouter 2026-04-06 04:31:55,508 - INFO - vrouter-nodemgr:active 2026-04-06 04:31:55,512 - INFO - agent:active 2026-04-06 04:31:55,601 - INFO - 10.0.0.241 2026-04-06 04:31:55,601 - INFO - vrouter 2026-04-06 04:31:55,606 - INFO - vrouter-nodemgr:active 2026-04-06 04:31:55,612 - INFO - agent:active 2026-04-06 04:31:55,613 - INFO - All the contrail services are up on all nodes 2026-04-06 04:31:55,613 - INFO - contrail-status passed 2026-04-06 04:31:56,887 - DEBUG - Skipping xmpp flap check 2026-04-06 04:31:56,887 - INFO - END TEST : test_contrail_status : PASSED[0:00:03] 2026-04-06 04:31:56,887 - INFO - --------------------------------------------------------------------------------