2026-09-28 02:41:52,199 - INFO - Using existing project ['default-domain', 'admin'](d4f6631a-2b5e-4b78-8aa6-54dc84bf6c57)
2026-09-28 02:41:52,217 - INFO - ================================================================================
2026-09-28 02:41:52,217 - INFO - STARTING TEST : test_contrail_status
2026-09-28 02:41:52,217 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active
2026-09-28 02:41:52,891 - DEBUG - Nothing to compare xmpp stats {'10.0.0.38': {'10.20.0.254': '0', '10.20.0.25': '0'}, '10.0.0.162': {'10.20.0.25': '0', '10.20.0.254': '0'}} with
2026-09-28 02:41:52,891 - INFO - Initial checks done. Running the testcase now
2026-09-28 02:41:52,891 - INFO -
2026-09-28 02:41:52,984 - INFO - 10.0.0.176
2026-09-28 02:41:52,984 - INFO - config
2026-09-28 02:41:52,994 - INFO - config-nodemgr:active
2026-09-28 02:41:53,001 - INFO - api-server:active
2026-09-28 02:41:53,005 - INFO - schema:backup
2026-09-28 02:41:53,009 - INFO - svc-monitor:backup
2026-09-28 02:41:53,012 - INFO - device-manager:backup
2026-09-28 02:41:53,012 - INFO - config-database
2026-09-28 02:41:53,012 - INFO - config-cassandra:active
2026-09-28 02:41:53,012 - INFO - config-zookeeper:active
2026-09-28 02:41:53,012 - INFO - config-rabbitmq:active
2026-09-28 02:41:53,012 - INFO - control
2026-09-28 02:41:53,018 - INFO - control-nodemgr:active
2026-09-28 02:41:53,028 - INFO - control:active
2026-09-28 02:41:53,028 - INFO - named:active
2026-09-28 02:41:53,036 - INFO - dns:active
2026-09-28 02:41:53,036 - INFO - analytics
2026-09-28 02:41:53,042 - INFO - analytics-nodemgr:active
2026-09-28 02:41:53,048 - INFO - analytics-api:active
2026-09-28 02:41:53,054 - INFO - collector:active
2026-09-28 02:41:53,054 - INFO - analytics-database
2026-09-28 02:41:53,055 - INFO - analytics-cassandra:active
2026-09-28 02:41:53,060 - INFO - analyticsdb-nodemgr:active
2026-09-28 02:41:53,070 - INFO - query-engine:active
2026-09-28 02:41:53,070 - INFO - webui
2026-09-28 02:41:53,071 - INFO - webui:active
2026-09-28 02:41:53,071 - INFO - webui-middleware:active
2026-09-28 02:41:53,071 - INFO - redis:active
2026-09-28 02:41:53,071 - INFO - analytics_snmp
2026-09-28 02:41:53,076 - INFO - snmp-collector:active
2026-09-28 02:41:53,076 - INFO - snmp-topology:active
2026-09-28 02:41:53,076 - INFO - analytics_alarm
2026-09-28 02:41:53,076 - INFO - alarmgen:active
2026-09-28 02:41:53,132 - INFO - 10.0.0.177
2026-09-28 02:41:53,132 - INFO - config
2026-09-28 02:41:53,140 - INFO - config-nodemgr:active
2026-09-28 02:41:53,145 - INFO - api-server:active
2026-09-28 02:41:53,151 - INFO - schema:active
2026-09-28 02:41:53,156 - INFO - svc-monitor:active
2026-09-28 02:41:53,161 - INFO - device-manager:active
2026-09-28 02:41:53,161 - INFO - config-database
2026-09-28 02:41:53,161 - INFO - config-cassandra:active
2026-09-28 02:41:53,161 - INFO - config-zookeeper:active
2026-09-28 02:41:53,161 - INFO - config-rabbitmq:active
2026-09-28 02:41:53,161 - INFO - control
2026-09-28 02:41:53,167 - INFO - control-nodemgr:active
2026-09-28 02:41:53,172 - INFO - control:active
2026-09-28 02:41:53,173 - INFO - named:active
2026-09-28 02:41:53,179 - INFO - dns:active
2026-09-28 02:41:53,179 - INFO - analytics
2026-09-28 02:41:53,185 - INFO - analytics-nodemgr:active
2026-09-28 02:41:53,191 - INFO - analytics-api:active
2026-09-28 02:41:53,197 - INFO - collector:active
2026-09-28 02:41:53,198 - INFO - analytics-database
2026-09-28 02:41:53,198 - INFO - analytics-cassandra:active
2026-09-28 02:41:53,203 - INFO - analyticsdb-nodemgr:active
2026-09-28 02:41:53,211 - INFO - query-engine:active
2026-09-28 02:41:53,211 - INFO - webui
2026-09-28 02:41:53,211 - INFO - webui:active
2026-09-28 02:41:53,211 - INFO - webui-middleware:active
2026-09-28 02:41:53,212 - INFO - redis:active
2026-09-28 02:41:53,212 - INFO - analytics_snmp
2026-09-28 02:41:53,217 - INFO - snmp-collector:active
2026-09-28 02:41:53,217 - INFO - snmp-topology:active
2026-09-28 02:41:53,217 - INFO - analytics_alarm
2026-09-28 02:41:53,217 - INFO - alarmgen:active
2026-09-28 02:41:53,284 - INFO - 10.0.0.161
2026-09-28 02:41:53,284 - INFO - config
2026-09-28 02:41:53,291 - INFO - config-nodemgr:active
2026-09-28 02:41:53,297 - INFO - api-server:active
2026-09-28 02:41:53,301 - INFO - schema:backup
2026-09-28 02:41:53,305 - INFO - svc-monitor:backup
2026-09-28 02:41:53,311 - INFO - device-manager:backup
2026-09-28 02:41:53,311 - INFO - config-database
2026-09-28 02:41:53,311 - INFO - config-cassandra:active
2026-09-28 02:41:53,311 - INFO - config-zookeeper:active
2026-09-28 02:41:53,311 - INFO - config-rabbitmq:active
2026-09-28 02:41:53,311 - INFO - control
2026-09-28 02:41:53,319 - INFO - control-nodemgr:active
2026-09-28 02:41:53,325 - INFO - control:active
2026-09-28 02:41:53,325 - INFO - named:active
2026-09-28 02:41:53,331 - INFO - dns:active
2026-09-28 02:41:53,331 - INFO - analytics
2026-09-28 02:41:53,338 - INFO - analytics-nodemgr:active
2026-09-28 02:41:53,344 - INFO - analytics-api:active
2026-09-28 02:41:53,351 - INFO - collector:active
2026-09-28 02:41:53,351 - INFO - analytics-database
2026-09-28 02:41:53,351 - INFO - analytics-cassandra:active
2026-09-28 02:41:53,357 - INFO - analyticsdb-nodemgr:active
2026-09-28 02:41:53,362 - INFO - query-engine:active
2026-09-28 02:41:53,363 - INFO - webui
2026-09-28 02:41:53,363 - INFO - webui:active
2026-09-28 02:41:53,363 - INFO - webui-middleware:active
2026-09-28 02:41:53,363 - INFO - redis:active
2026-09-28 02:41:53,363 - INFO - analytics_snmp
2026-09-28 02:41:53,368 - INFO - snmp-collector:active
2026-09-28 02:41:53,369 - INFO - snmp-topology:active
2026-09-28 02:41:53,369 - INFO - analytics_alarm
2026-09-28 02:41:53,369 - INFO - alarmgen:active
2026-09-28 02:41:53,426 - INFO - 10.0.0.38
2026-09-28 02:41:53,426 - INFO - vrouter
2026-09-28 02:41:53,431 - INFO - vrouter-nodemgr:active
2026-09-28 02:41:53,437 - INFO - agent:active
2026-09-28 02:41:53,482 - INFO - 10.0.0.162
2026-09-28 02:41:53,482 - INFO - vrouter
2026-09-28 02:41:53,489 - INFO - vrouter-nodemgr:active
2026-09-28 02:41:53,496 - INFO - agent:active
2026-09-28 02:41:53,496 - INFO - All the contrail services are up on all nodes
2026-09-28 02:41:53,496 - INFO - contrail-status passed
2026-09-28 02:41:54,169 - DEBUG - No XMPP flaps were noticed during the test
2026-09-28 02:41:54,170 - INFO - END TEST : test_contrail_status : PASSED[0:00:02]
2026-09-28 02:41:54,170 - INFO - --------------------------------------------------------------------------------
2026-09-28 05:02:51,195 - INFO - Using existing project ['default-domain', 'admin'](d4f6631a-2b5e-4b78-8aa6-54dc84bf6c57)
2026-09-28 05:02:51,211 - INFO - ================================================================================
2026-09-28 05:02:51,211 - INFO - STARTING TEST : test_contrail_status
2026-09-28 05:02:51,212 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active
2026-09-28 05:02:52,538 - DEBUG - Skipping xmpp flap check
2026-09-28 05:02:52,538 - INFO - Initial checks done. Running the testcase now
2026-09-28 05:02:52,538 - INFO -
2026-09-28 05:02:52,836 - INFO - 10.0.0.176
2026-09-28 05:02:52,836 - INFO - config
2026-09-28 05:02:52,846 - INFO - config-nodemgr:active
2026-09-28 05:02:52,856 - INFO - api-server:active
2026-09-28 05:02:52,866 - INFO - schema:backup
2026-09-28 05:02:52,884 - INFO - svc-monitor:backup
2026-09-28 05:02:52,902 - INFO - device-manager:backup
2026-09-28 05:02:52,902 - INFO - config-database
2026-09-28 05:02:52,902 - INFO - config-cassandra:active
2026-09-28 05:02:52,902 - INFO - config-zookeeper:active
2026-09-28 05:02:52,902 - INFO - config-rabbitmq:active
2026-09-28 05:02:52,902 - INFO - control
2026-09-28 05:02:52,911 - INFO - control-nodemgr:active
2026-09-28 05:02:52,923 - INFO - control:active
2026-09-28 05:02:52,924 - INFO - named:active
2026-09-28 05:02:52,945 - INFO - dns:active
2026-09-28 05:02:52,945 - INFO - analytics
2026-09-28 05:02:52,957 - INFO - analytics-nodemgr:active
2026-09-28 05:02:52,966 - INFO - analytics-api:active
2026-09-28 05:02:52,994 - INFO - collector:active
2026-09-28 05:02:52,996 - INFO - analytics-database
2026-09-28 05:02:52,997 - INFO - analytics-cassandra:active
2026-09-28 05:02:53,007 - INFO - analyticsdb-nodemgr:active
2026-09-28 05:02:53,031 - INFO - query-engine:active
2026-09-28 05:02:53,034 - INFO - webui
2026-09-28 05:02:53,034 - INFO - webui:active
2026-09-28 05:02:53,035 - INFO - webui-middleware:active
2026-09-28 05:02:53,035 - INFO - redis:active
2026-09-28 05:02:53,035 - INFO - analytics_snmp
2026-09-28 05:02:53,049 - INFO - snmp-collector:initializing (Database:RabbitMQ[] connection down)
2026-09-28 05:02:53,050 - INFO - snmp-topology:active
2026-09-28 05:02:53,051 - INFO - analytics_alarm
2026-09-28 05:02:53,051 - INFO - alarmgen:active
2026-09-28 05:02:53,163 - INFO - 10.0.0.177
2026-09-28 05:02:53,164 - INFO - config
2026-09-28 05:02:53,171 - INFO - config-nodemgr:active
2026-09-28 05:02:53,177 - INFO - api-server:active
2026-09-28 05:02:53,185 - INFO - schema:active
2026-09-28 05:02:53,192 - INFO - svc-monitor:active
2026-09-28 05:02:53,199 - INFO - device-manager:active
2026-09-28 05:02:53,199 - INFO - config-database
2026-09-28 05:02:53,199 - INFO - config-cassandra:active
2026-09-28 05:02:53,199 - INFO - config-zookeeper:active
2026-09-28 05:02:53,199 - INFO - config-rabbitmq:active
2026-09-28 05:02:53,199 - INFO - control
2026-09-28 05:02:53,207 - INFO - control-nodemgr:active
2026-09-28 05:02:53,226 - INFO - control:active
2026-09-28 05:02:53,226 - INFO - named:active
2026-09-28 05:02:53,234 - INFO - dns:active
2026-09-28 05:02:53,235 - INFO - analytics
2026-09-28 05:02:53,243 - INFO - analytics-nodemgr:active
2026-09-28 05:02:53,250 - INFO - analytics-api:active
2026-09-28 05:02:53,257 - INFO - collector:active
2026-09-28 05:02:53,257 - INFO - analytics-database
2026-09-28 05:02:53,257 - INFO - analytics-cassandra:active
2026-09-28 05:02:53,264 - INFO - analyticsdb-nodemgr:active
2026-09-28 05:02:53,271 - INFO - query-engine:active
2026-09-28 05:02:53,271 - INFO - webui
2026-09-28 05:02:53,272 - INFO - webui:active
2026-09-28 05:02:53,272 - INFO - webui-middleware:active
2026-09-28 05:02:53,272 - INFO - redis:active
2026-09-28 05:02:53,272 - INFO - analytics_snmp
2026-09-28 05:02:53,278 - INFO - snmp-collector:active
2026-09-28 05:02:53,278 - INFO - snmp-topology:active
2026-09-28 05:02:53,278 - INFO - analytics_alarm
2026-09-28 05:02:53,278 - INFO - alarmgen:active
2026-09-28 05:02:53,380 - INFO - 10.0.0.161
2026-09-28 05:02:53,380 - INFO - config
2026-09-28 05:02:53,389 - INFO - config-nodemgr:active
2026-09-28 05:02:53,396 - INFO - api-server:active
2026-09-28 05:02:53,401 - INFO - schema:backup
2026-09-28 05:02:53,404 - INFO - svc-monitor:backup
2026-09-28 05:02:53,407 - INFO - device-manager:backup
2026-09-28 05:02:53,408 - INFO - config-database
2026-09-28 05:02:53,408 - INFO - config-cassandra:active
2026-09-28 05:02:53,408 - INFO - config-zookeeper:active
2026-09-28 05:02:53,408 - INFO - config-rabbitmq:active
2026-09-28 05:02:53,408 - INFO - control
2026-09-28 05:02:53,413 - INFO - control-nodemgr:active
2026-09-28 05:02:53,419 - INFO - control:active
2026-09-28 05:02:53,419 - INFO - named:active
2026-09-28 05:02:53,427 - INFO - dns:active
2026-09-28 05:02:53,427 - INFO - analytics
2026-09-28 05:02:53,435 - INFO - analytics-nodemgr:active
2026-09-28 05:02:53,443 - INFO - analytics-api:active
2026-09-28 05:02:53,452 - INFO - collector:active
2026-09-28 05:02:53,452 - INFO - analytics-database
2026-09-28 05:02:53,452 - INFO - analytics-cassandra:active
2026-09-28 05:02:53,459 - INFO - analyticsdb-nodemgr:active
2026-09-28 05:02:53,467 - INFO - query-engine:active
2026-09-28 05:02:53,467 - INFO - webui
2026-09-28 05:02:53,467 - INFO - webui:active
2026-09-28 05:02:53,467 - INFO - webui-middleware:active
2026-09-28 05:02:53,467 - INFO - redis:active
2026-09-28 05:02:53,467 - INFO - analytics_snmp
2026-09-28 05:02:53,474 - INFO - snmp-collector:active
2026-09-28 05:02:53,474 - INFO - snmp-topology:active
2026-09-28 05:02:53,474 - INFO - analytics_alarm
2026-09-28 05:02:53,474 - INFO - alarmgen:active
2026-09-28 05:02:53,592 - INFO - 10.0.0.38
2026-09-28 05:02:53,592 - INFO - vrouter
2026-09-28 05:02:53,602 - INFO - vrouter-nodemgr:active
2026-09-28 05:02:53,610 - INFO - agent:active
2026-09-28 05:02:53,710 - INFO - 10.0.0.162
2026-09-28 05:02:53,710 - INFO - vrouter
2026-09-28 05:02:53,718 - INFO - vrouter-nodemgr:active
2026-09-28 05:02:53,725 - INFO - agent:active
2026-09-28 05:02:53,726 - DEBUG - defaultdict(, {'10.0.0.176': {'snmp-collector': {'status': 'initializing', 'description': 'Database:RabbitMQ[] connection down'}}})
2026-09-28 05:02:53,726 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 0
2026-09-28 05:03:03,864 - INFO - 10.0.0.176
2026-09-28 05:03:03,864 - INFO - config
2026-09-28 05:03:03,871 - INFO - config-nodemgr:active
2026-09-28 05:03:03,878 - INFO - api-server:active
2026-09-28 05:03:03,883 - INFO - schema:backup
2026-09-28 05:03:03,887 - INFO - svc-monitor:backup
2026-09-28 05:03:03,892 - INFO - device-manager:backup
2026-09-28 05:03:03,892 - INFO - config-database
2026-09-28 05:03:03,892 - INFO - config-cassandra:active
2026-09-28 05:03:03,892 - INFO - config-zookeeper:active
2026-09-28 05:03:03,892 - INFO - config-rabbitmq:active
2026-09-28 05:03:03,892 - INFO - control
2026-09-28 05:03:03,900 - INFO - control-nodemgr:active
2026-09-28 05:03:03,909 - INFO - control:active
2026-09-28 05:03:03,909 - INFO - named:active
2026-09-28 05:03:03,920 - INFO - dns:active
2026-09-28 05:03:03,920 - INFO - analytics
2026-09-28 05:03:03,930 - INFO - analytics-nodemgr:active
2026-09-28 05:03:03,937 - INFO - analytics-api:active
2026-09-28 05:03:03,944 - INFO - collector:active
2026-09-28 05:03:03,944 - INFO - analytics-database
2026-09-28 05:03:03,944 - INFO - analytics-cassandra:active
2026-09-28 05:03:03,951 - INFO - analyticsdb-nodemgr:active
2026-09-28 05:03:03,962 - INFO - query-engine:active
2026-09-28 05:03:03,962 - INFO - webui
2026-09-28 05:03:03,962 - INFO - webui:active
2026-09-28 05:03:03,962 - INFO - webui-middleware:active
2026-09-28 05:03:03,962 - INFO - redis:active
2026-09-28 05:03:03,962 - INFO - analytics_snmp
2026-09-28 05:03:03,968 - INFO - snmp-collector:active
2026-09-28 05:03:03,969 - INFO - snmp-topology:active
2026-09-28 05:03:03,969 - INFO - analytics_alarm
2026-09-28 05:03:03,969 - INFO - alarmgen:active
2026-09-28 05:03:04,072 - INFO - 10.0.0.177
2026-09-28 05:03:04,072 - INFO - config
2026-09-28 05:03:04,079 - INFO - config-nodemgr:active
2026-09-28 05:03:04,086 - INFO - api-server:active
2026-09-28 05:03:04,092 - INFO - schema:active
2026-09-28 05:03:04,099 - INFO - svc-monitor:active
2026-09-28 05:03:04,106 - INFO - device-manager:active
2026-09-28 05:03:04,106 - INFO - config-database
2026-09-28 05:03:04,106 - INFO - config-cassandra:active
2026-09-28 05:03:04,106 - INFO - config-zookeeper:active
2026-09-28 05:03:04,107 - INFO - config-rabbitmq:active
2026-09-28 05:03:04,107 - INFO - control
2026-09-28 05:03:04,115 - INFO - control-nodemgr:active
2026-09-28 05:03:04,123 - INFO - control:active
2026-09-28 05:03:04,123 - INFO - named:active
2026-09-28 05:03:04,131 - INFO - dns:active
2026-09-28 05:03:04,131 - INFO - analytics
2026-09-28 05:03:04,140 - INFO - analytics-nodemgr:active
2026-09-28 05:03:04,147 - INFO - analytics-api:active
2026-09-28 05:03:04,154 - INFO - collector:active
2026-09-28 05:03:04,154 - INFO - analytics-database
2026-09-28 05:03:04,155 - INFO - analytics-cassandra:active
2026-09-28 05:03:04,162 - INFO - analyticsdb-nodemgr:active
2026-09-28 05:03:04,170 - INFO - query-engine:active
2026-09-28 05:03:04,171 - INFO - webui
2026-09-28 05:03:04,171 - INFO - webui:active
2026-09-28 05:03:04,171 - INFO - webui-middleware:active
2026-09-28 05:03:04,171 - INFO - redis:active
2026-09-28 05:03:04,171 - INFO - analytics_snmp
2026-09-28 05:03:04,179 - INFO - snmp-collector:active
2026-09-28 05:03:04,179 - INFO - snmp-topology:active
2026-09-28 05:03:04,179 - INFO - analytics_alarm
2026-09-28 05:03:04,179 - INFO - alarmgen:active
2026-09-28 05:03:04,282 - INFO - 10.0.0.161
2026-09-28 05:03:04,282 - INFO - config
2026-09-28 05:03:04,287 - INFO - config-nodemgr:active
2026-09-28 05:03:04,295 - INFO - api-server:active
2026-09-28 05:03:04,300 - INFO - schema:backup
2026-09-28 05:03:04,304 - INFO - svc-monitor:backup
2026-09-28 05:03:04,310 - INFO - device-manager:backup
2026-09-28 05:03:04,310 - INFO - config-database
2026-09-28 05:03:04,310 - INFO - config-cassandra:active
2026-09-28 05:03:04,310 - INFO - config-zookeeper:active
2026-09-28 05:03:04,310 - INFO - config-rabbitmq:active
2026-09-28 05:03:04,310 - INFO - control
2026-09-28 05:03:04,317 - INFO - control-nodemgr:active
2026-09-28 05:03:04,328 - INFO - control:active
2026-09-28 05:03:04,329 - INFO - named:active
2026-09-28 05:03:04,337 - INFO - dns:active
2026-09-28 05:03:04,338 - INFO - analytics
2026-09-28 05:03:04,349 - INFO - analytics-nodemgr:active
2026-09-28 05:03:04,356 - INFO - analytics-api:active
2026-09-28 05:03:04,365 - INFO - collector:active
2026-09-28 05:03:04,365 - INFO - analytics-database
2026-09-28 05:03:04,365 - INFO - analytics-cassandra:active
2026-09-28 05:03:04,372 - INFO - analyticsdb-nodemgr:active
2026-09-28 05:03:04,379 - INFO - query-engine:active
2026-09-28 05:03:04,379 - INFO - webui
2026-09-28 05:03:04,380 - INFO - webui:active
2026-09-28 05:03:04,380 - INFO - webui-middleware:active
2026-09-28 05:03:04,380 - INFO - redis:active
2026-09-28 05:03:04,380 - INFO - analytics_snmp
2026-09-28 05:03:04,387 - INFO - snmp-collector:active
2026-09-28 05:03:04,387 - INFO - snmp-topology:active
2026-09-28 05:03:04,387 - INFO - analytics_alarm
2026-09-28 05:03:04,387 - INFO - alarmgen:active
2026-09-28 05:03:04,499 - INFO - 10.0.0.38
2026-09-28 05:03:04,499 - INFO - vrouter
2026-09-28 05:03:04,508 - INFO - vrouter-nodemgr:active
2026-09-28 05:03:04,516 - INFO - agent:active
2026-09-28 05:03:04,617 - INFO - 10.0.0.162
2026-09-28 05:03:04,618 - INFO - vrouter
2026-09-28 05:03:04,628 - INFO - vrouter-nodemgr:active
2026-09-28 05:03:04,642 - INFO - agent:active
2026-09-28 05:03:04,642 - INFO - All the contrail services are up on all nodes
2026-09-28 05:03:04,643 - INFO - contrail-status passed
2026-09-28 05:03:05,994 - DEBUG - Skipping xmpp flap check
2026-09-28 05:03:05,994 - INFO - END TEST : test_contrail_status : PASSED[0:00:14]
2026-09-28 05:03:05,994 - INFO - --------------------------------------------------------------------------------