2025-11-27 02:09:06,662 - INFO - Using existing project ['default-domain', 'admin'](2cb45a1e-c495-4d45-8a74-dee6721f0cec)
2025-11-27 02:09:06,671 - INFO - ================================================================================
2025-11-27 02:09:06,671 - INFO - STARTING TEST : test_contrail_status
2025-11-27 02:09:06,671 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active
2025-11-27 02:09:07,305 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.193': '0', '10.20.0.192': '0'}, '10.0.0.130': {'10.20.0.192': '0', '10.20.0.129': '0'}} with
2025-11-27 02:09:07,305 - INFO - Initial checks done. Running the testcase now
2025-11-27 02:09:07,305 - INFO -
2025-11-27 02:09:07,384 - INFO - 10.0.0.27
2025-11-27 02:09:07,385 - INFO - config
2025-11-27 02:09:07,389 - INFO - config-nodemgr:active
2025-11-27 02:09:07,394 - INFO - api-server:active
2025-11-27 02:09:07,398 - INFO - schema:active
2025-11-27 02:09:07,401 - INFO - svc-monitor:backup
2025-11-27 02:09:07,403 - INFO - device-manager:backup
2025-11-27 02:09:07,403 - INFO - config-database
2025-11-27 02:09:07,404 - INFO - config-cassandra:active
2025-11-27 02:09:07,404 - INFO - config-zookeeper:active
2025-11-27 02:09:07,404 - INFO - config-rabbitmq:active
2025-11-27 02:09:07,404 - INFO - control
2025-11-27 02:09:07,408 - INFO - control-nodemgr:active
2025-11-27 02:09:07,413 - INFO - control:active
2025-11-27 02:09:07,413 - INFO - named:active
2025-11-27 02:09:07,419 - INFO - dns:active
2025-11-27 02:09:07,419 - INFO - analytics
2025-11-27 02:09:07,424 - INFO - analytics-nodemgr:active
2025-11-27 02:09:07,428 - INFO - analytics-api:active
2025-11-27 02:09:07,432 - INFO - collector:active
2025-11-27 02:09:07,432 - INFO - analytics-database
2025-11-27 02:09:07,433 - INFO - analytics-cassandra:active
2025-11-27 02:09:07,438 - INFO - analyticsdb-nodemgr:active
2025-11-27 02:09:07,444 - INFO - query-engine:active
2025-11-27 02:09:07,444 - INFO - webui
2025-11-27 02:09:07,444 - INFO - webui:active
2025-11-27 02:09:07,445 - INFO - webui-middleware:active
2025-11-27 02:09:07,445 - INFO - redis:active
2025-11-27 02:09:07,445 - INFO - analytics_snmp
2025-11-27 02:09:07,449 - INFO - snmp-collector:active
2025-11-27 02:09:07,449 - INFO - snmp-topology:active
2025-11-27 02:09:07,449 - INFO - analytics_alarm
2025-11-27 02:09:07,449 - INFO - alarmgen:active
2025-11-27 02:09:07,508 - INFO - 10.0.0.33
2025-11-27 02:09:07,508 - INFO - config
2025-11-27 02:09:07,514 - INFO - config-nodemgr:active
2025-11-27 02:09:07,519 - INFO - api-server:active
2025-11-27 02:09:07,522 - INFO - schema:backup
2025-11-27 02:09:07,526 - INFO - svc-monitor:backup
2025-11-27 02:09:07,529 - INFO - device-manager:backup
2025-11-27 02:09:07,529 - INFO - config-database
2025-11-27 02:09:07,529 - INFO - config-cassandra:active
2025-11-27 02:09:07,529 - INFO - config-zookeeper:active
2025-11-27 02:09:07,529 - INFO - config-rabbitmq:active
2025-11-27 02:09:07,530 - INFO - control
2025-11-27 02:09:07,535 - INFO - control-nodemgr:active
2025-11-27 02:09:07,540 - INFO - control:active
2025-11-27 02:09:07,540 - INFO - named:active
2025-11-27 02:09:07,545 - INFO - dns:active
2025-11-27 02:09:07,545 - INFO - analytics
2025-11-27 02:09:07,550 - INFO - analytics-nodemgr:active
2025-11-27 02:09:07,554 - INFO - analytics-api:active
2025-11-27 02:09:07,560 - INFO - collector:active
2025-11-27 02:09:07,560 - INFO - analytics-database
2025-11-27 02:09:07,560 - INFO - analytics-cassandra:active
2025-11-27 02:09:07,565 - INFO - analyticsdb-nodemgr:active
2025-11-27 02:09:07,571 - INFO - query-engine:active
2025-11-27 02:09:07,571 - INFO - webui
2025-11-27 02:09:07,571 - INFO - webui:active
2025-11-27 02:09:07,571 - INFO - webui-middleware:active
2025-11-27 02:09:07,571 - INFO - redis:active
2025-11-27 02:09:07,571 - INFO - analytics_snmp
2025-11-27 02:09:07,576 - INFO - snmp-collector:active
2025-11-27 02:09:07,576 - INFO - snmp-topology:active
2025-11-27 02:09:07,576 - INFO - analytics_alarm
2025-11-27 02:09:07,576 - INFO - alarmgen:active
2025-11-27 02:09:07,620 - INFO - 10.0.0.23
2025-11-27 02:09:07,621 - INFO - config
2025-11-27 02:09:07,626 - INFO - config-nodemgr:active
2025-11-27 02:09:07,632 - INFO - api-server:active
2025-11-27 02:09:07,635 - INFO - schema:backup
2025-11-27 02:09:07,639 - INFO - svc-monitor:active
2025-11-27 02:09:07,643 - INFO - device-manager:active
2025-11-27 02:09:07,643 - INFO - config-database
2025-11-27 02:09:07,643 - INFO - config-cassandra:active
2025-11-27 02:09:07,643 - INFO - config-zookeeper:active
2025-11-27 02:09:07,643 - INFO - config-rabbitmq:active
2025-11-27 02:09:07,643 - INFO - control
2025-11-27 02:09:07,649 - INFO - control-nodemgr:active
2025-11-27 02:09:07,655 - INFO - control:active
2025-11-27 02:09:07,655 - INFO - named:active
2025-11-27 02:09:07,661 - INFO - dns:active
2025-11-27 02:09:07,661 - INFO - analytics
2025-11-27 02:09:07,666 - INFO - analytics-nodemgr:active
2025-11-27 02:09:07,670 - INFO - analytics-api:active
2025-11-27 02:09:07,675 - INFO - collector:active
2025-11-27 02:09:07,675 - INFO - analytics-database
2025-11-27 02:09:07,675 - INFO - analytics-cassandra:active
2025-11-27 02:09:07,680 - INFO - analyticsdb-nodemgr:active
2025-11-27 02:09:07,685 - INFO - query-engine:active
2025-11-27 02:09:07,685 - INFO - webui
2025-11-27 02:09:07,685 - INFO - webui:active
2025-11-27 02:09:07,685 - INFO - webui-middleware:active
2025-11-27 02:09:07,685 - INFO - redis:active
2025-11-27 02:09:07,685 - INFO - analytics_snmp
2025-11-27 02:09:07,689 - INFO - snmp-collector:active
2025-11-27 02:09:07,689 - INFO - snmp-topology:active
2025-11-27 02:09:07,689 - INFO - analytics_alarm
2025-11-27 02:09:07,689 - INFO - alarmgen:active
2025-11-27 02:09:07,735 - INFO - 10.0.0.19
2025-11-27 02:09:07,735 - INFO - vrouter
2025-11-27 02:09:07,740 - INFO - vrouter-nodemgr:active
2025-11-27 02:09:07,746 - INFO - agent:active
2025-11-27 02:09:07,801 - INFO - 10.0.0.130
2025-11-27 02:09:07,801 - INFO - vrouter
2025-11-27 02:09:07,806 - INFO - vrouter-nodemgr:active
2025-11-27 02:09:07,813 - INFO - agent:active
2025-11-27 02:09:07,813 - INFO - All the contrail services are up on all nodes
2025-11-27 02:09:07,813 - INFO - contrail-status passed
2025-11-27 02:09:08,434 - DEBUG - No XMPP flaps were noticed during the test
2025-11-27 02:09:08,434 - INFO - END TEST : test_contrail_status : PASSED[0:00:02]
2025-11-27 02:09:08,434 - INFO - --------------------------------------------------------------------------------
2025-11-27 04:01:45,912 - INFO - Using existing project ['default-domain', 'admin'](2cb45a1e-c495-4d45-8a74-dee6721f0cec)
2025-11-27 04:01:45,929 - INFO - ================================================================================
2025-11-27 04:01:45,930 - INFO - STARTING TEST : test_contrail_status
2025-11-27 04:01:45,930 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active
2025-11-27 04:01:47,201 - DEBUG - Skipping xmpp flap check
2025-11-27 04:01:47,201 - INFO - Initial checks done. Running the testcase now
2025-11-27 04:01:47,202 - INFO -
2025-11-27 04:01:47,477 - INFO - 10.0.0.27
2025-11-27 04:01:47,477 - INFO - config
2025-11-27 04:01:47,483 - INFO - config-nodemgr:active
2025-11-27 04:01:47,487 - INFO - api-server:active
2025-11-27 04:01:47,491 - INFO - schema:active
2025-11-27 04:01:47,494 - INFO - svc-monitor:backup
2025-11-27 04:01:47,497 - INFO - device-manager:backup
2025-11-27 04:01:47,497 - INFO - config-database
2025-11-27 04:01:47,497 - INFO - config-cassandra:active
2025-11-27 04:01:47,497 - INFO - config-zookeeper:active
2025-11-27 04:01:47,497 - INFO - config-rabbitmq:active
2025-11-27 04:01:47,497 - INFO - control
2025-11-27 04:01:47,502 - INFO - control-nodemgr:active
2025-11-27 04:01:47,509 - INFO - control:active
2025-11-27 04:01:47,509 - INFO - named:active
2025-11-27 04:01:47,520 - INFO - dns:active
2025-11-27 04:01:47,520 - INFO - analytics
2025-11-27 04:01:47,524 - INFO - analytics-nodemgr:active
2025-11-27 04:01:47,530 - INFO - analytics-api:active
2025-11-27 04:01:47,553 - INFO - collector:active
2025-11-27 04:01:47,553 - INFO - analytics-database
2025-11-27 04:01:47,553 - INFO - analytics-cassandra:active
2025-11-27 04:01:47,558 - INFO - analyticsdb-nodemgr:active
2025-11-27 04:01:47,579 - INFO - query-engine:active
2025-11-27 04:01:47,579 - INFO - webui
2025-11-27 04:01:47,579 - INFO - webui:active
2025-11-27 04:01:47,579 - INFO - webui-middleware:active
2025-11-27 04:01:47,579 - INFO - redis:active
2025-11-27 04:01:47,579 - INFO - analytics_snmp
2025-11-27 04:01:47,583 - INFO - snmp-collector:active
2025-11-27 04:01:47,583 - INFO - snmp-topology:active
2025-11-27 04:01:47,583 - INFO - analytics_alarm
2025-11-27 04:01:47,583 - INFO - alarmgen:active
2025-11-27 04:01:47,694 - INFO - 10.0.0.33
2025-11-27 04:01:47,694 - INFO - config
2025-11-27 04:01:47,700 - INFO - config-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:01:47,705 - INFO - api-server:active
2025-11-27 04:01:47,708 - INFO - schema:backup
2025-11-27 04:01:47,711 - INFO - svc-monitor:backup
2025-11-27 04:01:47,714 - INFO - device-manager:backup
2025-11-27 04:01:47,714 - INFO - config-database
2025-11-27 04:01:47,714 - INFO - config-cassandra:active
2025-11-27 04:01:47,714 - INFO - config-zookeeper:active
2025-11-27 04:01:47,714 - INFO - config-rabbitmq:active
2025-11-27 04:01:47,714 - INFO - control
2025-11-27 04:01:47,719 - INFO - control-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:01:47,724 - INFO - control:active
2025-11-27 04:01:47,724 - INFO - named:active
2025-11-27 04:01:47,729 - INFO - dns:active
2025-11-27 04:01:47,729 - INFO - analytics
2025-11-27 04:01:47,734 - INFO - analytics-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:01:47,738 - INFO - analytics-api:active
2025-11-27 04:01:47,745 - INFO - collector:active
2025-11-27 04:01:47,745 - INFO - analytics-database
2025-11-27 04:01:47,745 - INFO - analytics-cassandra:active
2025-11-27 04:01:47,751 - INFO - analyticsdb-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:01:47,758 - INFO - query-engine:active
2025-11-27 04:01:47,758 - INFO - webui
2025-11-27 04:01:47,758 - INFO - webui:active
2025-11-27 04:01:47,758 - INFO - webui-middleware:active
2025-11-27 04:01:47,758 - INFO - redis:active
2025-11-27 04:01:47,758 - INFO - analytics_snmp
2025-11-27 04:01:47,762 - INFO - snmp-collector:active
2025-11-27 04:01:47,762 - INFO - snmp-topology:active
2025-11-27 04:01:47,763 - INFO - analytics_alarm
2025-11-27 04:01:47,763 - INFO - alarmgen:active
2025-11-27 04:01:47,851 - INFO - 10.0.0.23
2025-11-27 04:01:47,851 - INFO - config
2025-11-27 04:01:47,857 - INFO - config-nodemgr:active
2025-11-27 04:01:47,862 - INFO - api-server:active
2025-11-27 04:01:47,866 - INFO - schema:backup
2025-11-27 04:01:47,870 - INFO - svc-monitor:active
2025-11-27 04:01:47,875 - INFO - device-manager:active
2025-11-27 04:01:47,875 - INFO - config-database
2025-11-27 04:01:47,875 - INFO - config-cassandra:active
2025-11-27 04:01:47,875 - INFO - config-zookeeper:active
2025-11-27 04:01:47,875 - INFO - config-rabbitmq:active
2025-11-27 04:01:47,875 - INFO - control
2025-11-27 04:01:47,880 - INFO - control-nodemgr:active
2025-11-27 04:01:47,885 - INFO - control:active
2025-11-27 04:01:47,885 - INFO - named:active
2025-11-27 04:01:47,891 - INFO - dns:active
2025-11-27 04:01:47,891 - INFO - analytics
2025-11-27 04:01:47,895 - INFO - analytics-nodemgr:active
2025-11-27 04:01:47,899 - INFO - analytics-api:active
2025-11-27 04:01:47,904 - INFO - collector:active
2025-11-27 04:01:47,904 - INFO - analytics-database
2025-11-27 04:01:47,904 - INFO - analytics-cassandra:active
2025-11-27 04:01:47,908 - INFO - analyticsdb-nodemgr:active
2025-11-27 04:01:47,913 - INFO - query-engine:active
2025-11-27 04:01:47,913 - INFO - webui
2025-11-27 04:01:47,913 - INFO - webui:active
2025-11-27 04:01:47,913 - INFO - webui-middleware:active
2025-11-27 04:01:47,913 - INFO - redis:active
2025-11-27 04:01:47,913 - INFO - analytics_snmp
2025-11-27 04:01:47,917 - INFO - snmp-collector:active
2025-11-27 04:01:47,917 - INFO - snmp-topology:active
2025-11-27 04:01:47,917 - INFO - analytics_alarm
2025-11-27 04:01:47,917 - INFO - alarmgen:active
2025-11-27 04:01:48,015 - INFO - 10.0.0.19
2025-11-27 04:01:48,015 - INFO - vrouter
2025-11-27 04:01:48,021 - INFO - vrouter-nodemgr:active
2025-11-27 04:01:48,027 - INFO - agent:active
2025-11-27 04:01:48,126 - INFO - 10.0.0.130
2025-11-27 04:01:48,126 - INFO - vrouter
2025-11-27 04:01:48,133 - INFO - vrouter-nodemgr:active
2025-11-27 04:01:48,140 - INFO - agent:active
2025-11-27 04:01:48,140 - DEBUG - defaultdict(, {'10.0.0.33': {'config-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'control-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'analytics-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'analyticsdb-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}}})
2025-11-27 04:01:48,140 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 0
2025-11-27 04:01:58,254 - INFO - 10.0.0.27
2025-11-27 04:01:58,254 - INFO - config
2025-11-27 04:01:58,258 - INFO - config-nodemgr:active
2025-11-27 04:01:58,263 - INFO - api-server:active
2025-11-27 04:01:58,266 - INFO - schema:active
2025-11-27 04:01:58,269 - INFO - svc-monitor:backup
2025-11-27 04:01:58,272 - INFO - device-manager:backup
2025-11-27 04:01:58,272 - INFO - config-database
2025-11-27 04:01:58,272 - INFO - config-cassandra:active
2025-11-27 04:01:58,272 - INFO - config-zookeeper:active
2025-11-27 04:01:58,272 - INFO - config-rabbitmq:active
2025-11-27 04:01:58,272 - INFO - control
2025-11-27 04:01:58,277 - INFO - control-nodemgr:active
2025-11-27 04:01:58,282 - INFO - control:active
2025-11-27 04:01:58,282 - INFO - named:active
2025-11-27 04:01:58,288 - INFO - dns:active
2025-11-27 04:01:58,288 - INFO - analytics
2025-11-27 04:01:58,293 - INFO - analytics-nodemgr:active
2025-11-27 04:01:58,297 - INFO - analytics-api:active
2025-11-27 04:01:58,303 - INFO - collector:active
2025-11-27 04:01:58,303 - INFO - analytics-database
2025-11-27 04:01:58,303 - INFO - analytics-cassandra:active
2025-11-27 04:01:58,308 - INFO - analyticsdb-nodemgr:active
2025-11-27 04:01:58,315 - INFO - query-engine:active
2025-11-27 04:01:58,316 - INFO - webui
2025-11-27 04:01:58,316 - INFO - webui:active
2025-11-27 04:01:58,316 - INFO - webui-middleware:active
2025-11-27 04:01:58,316 - INFO - redis:active
2025-11-27 04:01:58,316 - INFO - analytics_snmp
2025-11-27 04:01:58,319 - INFO - snmp-collector:active
2025-11-27 04:01:58,319 - INFO - snmp-topology:active
2025-11-27 04:01:58,319 - INFO - analytics_alarm
2025-11-27 04:01:58,319 - INFO - alarmgen:active
2025-11-27 04:01:58,420 - INFO - 10.0.0.33
2025-11-27 04:01:58,420 - INFO - config
2025-11-27 04:01:58,426 - INFO - config-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:01:58,431 - INFO - api-server:active
2025-11-27 04:01:58,435 - INFO - schema:backup
2025-11-27 04:01:58,441 - INFO - svc-monitor:backup
2025-11-27 04:01:58,445 - INFO - device-manager:backup
2025-11-27 04:01:58,445 - INFO - config-database
2025-11-27 04:01:58,445 - INFO - config-cassandra:active
2025-11-27 04:01:58,445 - INFO - config-zookeeper:active
2025-11-27 04:01:58,445 - INFO - config-rabbitmq:active
2025-11-27 04:01:58,445 - INFO - control
2025-11-27 04:01:58,452 - INFO - control-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:01:58,457 - INFO - control:active
2025-11-27 04:01:58,457 - INFO - named:active
2025-11-27 04:01:58,463 - INFO - dns:active
2025-11-27 04:01:58,463 - INFO - analytics
2025-11-27 04:01:58,471 - INFO - analytics-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:01:58,475 - INFO - analytics-api:active
2025-11-27 04:01:58,480 - INFO - collector:active
2025-11-27 04:01:58,480 - INFO - analytics-database
2025-11-27 04:01:58,480 - INFO - analytics-cassandra:active
2025-11-27 04:01:58,484 - INFO - analyticsdb-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:01:58,489 - INFO - query-engine:active
2025-11-27 04:01:58,489 - INFO - webui
2025-11-27 04:01:58,489 - INFO - webui:active
2025-11-27 04:01:58,489 - INFO - webui-middleware:active
2025-11-27 04:01:58,489 - INFO - redis:active
2025-11-27 04:01:58,489 - INFO - analytics_snmp
2025-11-27 04:01:58,493 - INFO - snmp-collector:active
2025-11-27 04:01:58,494 - INFO - snmp-topology:active
2025-11-27 04:01:58,494 - INFO - analytics_alarm
2025-11-27 04:01:58,494 - INFO - alarmgen:active
2025-11-27 04:01:58,592 - INFO - 10.0.0.23
2025-11-27 04:01:58,592 - INFO - config
2025-11-27 04:01:58,597 - INFO - config-nodemgr:active
2025-11-27 04:01:58,601 - INFO - api-server:active
2025-11-27 04:01:58,604 - INFO - schema:backup
2025-11-27 04:01:58,609 - INFO - svc-monitor:active
2025-11-27 04:01:58,613 - INFO - device-manager:active
2025-11-27 04:01:58,613 - INFO - config-database
2025-11-27 04:01:58,613 - INFO - config-cassandra:active
2025-11-27 04:01:58,613 - INFO - config-zookeeper:active
2025-11-27 04:01:58,613 - INFO - config-rabbitmq:active
2025-11-27 04:01:58,613 - INFO - control
2025-11-27 04:01:58,618 - INFO - control-nodemgr:active
2025-11-27 04:01:58,623 - INFO - control:active
2025-11-27 04:01:58,623 - INFO - named:active
2025-11-27 04:01:58,630 - INFO - dns:active
2025-11-27 04:01:58,630 - INFO - analytics
2025-11-27 04:01:58,635 - INFO - analytics-nodemgr:active
2025-11-27 04:01:58,639 - INFO - analytics-api:active
2025-11-27 04:01:58,644 - INFO - collector:active
2025-11-27 04:01:58,644 - INFO - analytics-database
2025-11-27 04:01:58,644 - INFO - analytics-cassandra:active
2025-11-27 04:01:58,649 - INFO - analyticsdb-nodemgr:active
2025-11-27 04:01:58,654 - INFO - query-engine:active
2025-11-27 04:01:58,654 - INFO - webui
2025-11-27 04:01:58,654 - INFO - webui:active
2025-11-27 04:01:58,654 - INFO - webui-middleware:active
2025-11-27 04:01:58,654 - INFO - redis:active
2025-11-27 04:01:58,654 - INFO - analytics_snmp
2025-11-27 04:01:58,658 - INFO - snmp-collector:active
2025-11-27 04:01:58,658 - INFO - snmp-topology:active
2025-11-27 04:01:58,658 - INFO - analytics_alarm
2025-11-27 04:01:58,658 - INFO - alarmgen:active
2025-11-27 04:01:58,745 - INFO - 10.0.0.19
2025-11-27 04:01:58,745 - INFO - vrouter
2025-11-27 04:01:58,750 - INFO - vrouter-nodemgr:active
2025-11-27 04:01:58,755 - INFO - agent:active
2025-11-27 04:01:58,843 - INFO - 10.0.0.130
2025-11-27 04:01:58,843 - INFO - vrouter
2025-11-27 04:01:58,848 - INFO - vrouter-nodemgr:active
2025-11-27 04:01:58,853 - INFO - agent:active
2025-11-27 04:01:58,853 - DEBUG - defaultdict(, {'10.0.0.33': {'config-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'control-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'analytics-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'analyticsdb-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}}})
2025-11-27 04:01:58,853 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 1
2025-11-27 04:02:08,963 - INFO - 10.0.0.27
2025-11-27 04:02:08,964 - INFO - config
2025-11-27 04:02:08,968 - INFO - config-nodemgr:active
2025-11-27 04:02:08,972 - INFO - api-server:active
2025-11-27 04:02:08,975 - INFO - schema:active
2025-11-27 04:02:08,978 - INFO - svc-monitor:backup
2025-11-27 04:02:08,980 - INFO - device-manager:backup
2025-11-27 04:02:08,980 - INFO - config-database
2025-11-27 04:02:08,980 - INFO - config-cassandra:active
2025-11-27 04:02:08,980 - INFO - config-zookeeper:active
2025-11-27 04:02:08,980 - INFO - config-rabbitmq:active
2025-11-27 04:02:08,980 - INFO - control
2025-11-27 04:02:08,986 - INFO - control-nodemgr:active
2025-11-27 04:02:08,992 - INFO - control:active
2025-11-27 04:02:08,992 - INFO - named:active
2025-11-27 04:02:08,996 - INFO - dns:active
2025-11-27 04:02:08,997 - INFO - analytics
2025-11-27 04:02:09,001 - INFO - analytics-nodemgr:active
2025-11-27 04:02:09,005 - INFO - analytics-api:active
2025-11-27 04:02:09,009 - INFO - collector:active
2025-11-27 04:02:09,009 - INFO - analytics-database
2025-11-27 04:02:09,009 - INFO - analytics-cassandra:active
2025-11-27 04:02:09,012 - INFO - analyticsdb-nodemgr:active
2025-11-27 04:02:09,016 - INFO - query-engine:active
2025-11-27 04:02:09,016 - INFO - webui
2025-11-27 04:02:09,016 - INFO - webui:active
2025-11-27 04:02:09,016 - INFO - webui-middleware:active
2025-11-27 04:02:09,016 - INFO - redis:active
2025-11-27 04:02:09,016 - INFO - analytics_snmp
2025-11-27 04:02:09,020 - INFO - snmp-collector:active
2025-11-27 04:02:09,020 - INFO - snmp-topology:active
2025-11-27 04:02:09,020 - INFO - analytics_alarm
2025-11-27 04:02:09,020 - INFO - alarmgen:active
2025-11-27 04:02:09,130 - INFO - 10.0.0.33
2025-11-27 04:02:09,130 - INFO - config
2025-11-27 04:02:09,135 - INFO - config-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:02:09,140 - INFO - api-server:active
2025-11-27 04:02:09,144 - INFO - schema:backup
2025-11-27 04:02:09,148 - INFO - svc-monitor:backup
2025-11-27 04:02:09,152 - INFO - device-manager:backup
2025-11-27 04:02:09,152 - INFO - config-database
2025-11-27 04:02:09,152 - INFO - config-cassandra:active
2025-11-27 04:02:09,152 - INFO - config-zookeeper:active
2025-11-27 04:02:09,152 - INFO - config-rabbitmq:active
2025-11-27 04:02:09,152 - INFO - control
2025-11-27 04:02:09,158 - INFO - control-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:02:09,163 - INFO - control:active
2025-11-27 04:02:09,163 - INFO - named:active
2025-11-27 04:02:09,168 - INFO - dns:active
2025-11-27 04:02:09,168 - INFO - analytics
2025-11-27 04:02:09,174 - INFO - analytics-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:02:09,179 - INFO - analytics-api:active
2025-11-27 04:02:09,185 - INFO - collector:active
2025-11-27 04:02:09,185 - INFO - analytics-database
2025-11-27 04:02:09,185 - INFO - analytics-cassandra:active
2025-11-27 04:02:09,189 - INFO - analyticsdb-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:02:09,195 - INFO - query-engine:active
2025-11-27 04:02:09,195 - INFO - webui
2025-11-27 04:02:09,195 - INFO - webui:active
2025-11-27 04:02:09,195 - INFO - webui-middleware:active
2025-11-27 04:02:09,195 - INFO - redis:active
2025-11-27 04:02:09,195 - INFO - analytics_snmp
2025-11-27 04:02:09,200 - INFO - snmp-collector:active
2025-11-27 04:02:09,200 - INFO - snmp-topology:active
2025-11-27 04:02:09,200 - INFO - analytics_alarm
2025-11-27 04:02:09,200 - INFO - alarmgen:active
2025-11-27 04:02:09,287 - INFO - 10.0.0.23
2025-11-27 04:02:09,287 - INFO - config
2025-11-27 04:02:09,292 - INFO - config-nodemgr:active
2025-11-27 04:02:09,297 - INFO - api-server:active
2025-11-27 04:02:09,300 - INFO - schema:backup
2025-11-27 04:02:09,304 - INFO - svc-monitor:active
2025-11-27 04:02:09,308 - INFO - device-manager:active
2025-11-27 04:02:09,308 - INFO - config-database
2025-11-27 04:02:09,308 - INFO - config-cassandra:active
2025-11-27 04:02:09,308 - INFO - config-zookeeper:active
2025-11-27 04:02:09,308 - INFO - config-rabbitmq:active
2025-11-27 04:02:09,308 - INFO - control
2025-11-27 04:02:09,312 - INFO - control-nodemgr:active
2025-11-27 04:02:09,316 - INFO - control:active
2025-11-27 04:02:09,316 - INFO - named:active
2025-11-27 04:02:09,321 - INFO - dns:active
2025-11-27 04:02:09,321 - INFO - analytics
2025-11-27 04:02:09,325 - INFO - analytics-nodemgr:active
2025-11-27 04:02:09,329 - INFO - analytics-api:active
2025-11-27 04:02:09,333 - INFO - collector:active
2025-11-27 04:02:09,333 - INFO - analytics-database
2025-11-27 04:02:09,333 - INFO - analytics-cassandra:active
2025-11-27 04:02:09,337 - INFO - analyticsdb-nodemgr:active
2025-11-27 04:02:09,342 - INFO - query-engine:active
2025-11-27 04:02:09,342 - INFO - webui
2025-11-27 04:02:09,342 - INFO - webui:active
2025-11-27 04:02:09,342 - INFO - webui-middleware:active
2025-11-27 04:02:09,342 - INFO - redis:active
2025-11-27 04:02:09,342 - INFO - analytics_snmp
2025-11-27 04:02:09,346 - INFO - snmp-collector:active
2025-11-27 04:02:09,346 - INFO - snmp-topology:active
2025-11-27 04:02:09,347 - INFO - analytics_alarm
2025-11-27 04:02:09,347 - INFO - alarmgen:active
2025-11-27 04:02:09,432 - INFO - 10.0.0.19
2025-11-27 04:02:09,432 - INFO - vrouter
2025-11-27 04:02:09,438 - INFO - vrouter-nodemgr:active
2025-11-27 04:02:09,444 - INFO - agent:active
2025-11-27 04:02:09,531 - INFO - 10.0.0.130
2025-11-27 04:02:09,531 - INFO - vrouter
2025-11-27 04:02:09,536 - INFO - vrouter-nodemgr:active
2025-11-27 04:02:09,541 - INFO - agent:active
2025-11-27 04:02:09,541 - DEBUG - defaultdict(, {'10.0.0.33': {'config-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'control-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'analytics-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'analyticsdb-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}}})
2025-11-27 04:02:09,541 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 2
2025-11-27 04:02:19,657 - INFO - 10.0.0.27
2025-11-27 04:02:19,657 - INFO - config
2025-11-27 04:02:19,662 - INFO - config-nodemgr:active
2025-11-27 04:02:19,667 - INFO - api-server:active
2025-11-27 04:02:19,671 - INFO - schema:active
2025-11-27 04:02:19,674 - INFO - svc-monitor:backup
2025-11-27 04:02:19,678 - INFO - device-manager:backup
2025-11-27 04:02:19,678 - INFO - config-database
2025-11-27 04:02:19,678 - INFO - config-cassandra:active
2025-11-27 04:02:19,678 - INFO - config-zookeeper:active
2025-11-27 04:02:19,678 - INFO - config-rabbitmq:active
2025-11-27 04:02:19,679 - INFO - control
2025-11-27 04:02:19,683 - INFO - control-nodemgr:active
2025-11-27 04:02:19,689 - INFO - control:active
2025-11-27 04:02:19,689 - INFO - named:active
2025-11-27 04:02:19,696 - INFO - dns:active
2025-11-27 04:02:19,696 - INFO - analytics
2025-11-27 04:02:19,701 - INFO - analytics-nodemgr:active
2025-11-27 04:02:19,706 - INFO - analytics-api:active
2025-11-27 04:02:19,713 - INFO - collector:active
2025-11-27 04:02:19,713 - INFO - analytics-database
2025-11-27 04:02:19,713 - INFO - analytics-cassandra:active
2025-11-27 04:02:19,717 - INFO - analyticsdb-nodemgr:active
2025-11-27 04:02:19,723 - INFO - query-engine:active
2025-11-27 04:02:19,723 - INFO - webui
2025-11-27 04:02:19,724 - INFO - webui:active
2025-11-27 04:02:19,724 - INFO - webui-middleware:active
2025-11-27 04:02:19,724 - INFO - redis:active
2025-11-27 04:02:19,724 - INFO - analytics_snmp
2025-11-27 04:02:19,727 - INFO - snmp-collector:active
2025-11-27 04:02:19,727 - INFO - snmp-topology:active
2025-11-27 04:02:19,727 - INFO - analytics_alarm
2025-11-27 04:02:19,727 - INFO - alarmgen:active
2025-11-27 04:02:19,829 - INFO - 10.0.0.33
2025-11-27 04:02:19,829 - INFO - config
2025-11-27 04:02:19,835 - INFO - config-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:02:19,840 - INFO - api-server:active
2025-11-27 04:02:19,844 - INFO - schema:backup
2025-11-27 04:02:19,847 - INFO - svc-monitor:backup
2025-11-27 04:02:19,851 - INFO - device-manager:backup
2025-11-27 04:02:19,851 - INFO - config-database
2025-11-27 04:02:19,851 - INFO - config-cassandra:active
2025-11-27 04:02:19,851 - INFO - config-zookeeper:active
2025-11-27 04:02:19,851 - INFO - config-rabbitmq:active
2025-11-27 04:02:19,851 - INFO - control
2025-11-27 04:02:19,856 - INFO - control-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:02:19,861 - INFO - control:active
2025-11-27 04:02:19,862 - INFO - named:active
2025-11-27 04:02:19,868 - INFO - dns:active
2025-11-27 04:02:19,868 - INFO - analytics
2025-11-27 04:02:19,873 - INFO - analytics-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:02:19,878 - INFO - analytics-api:active
2025-11-27 04:02:19,883 - INFO - collector:active
2025-11-27 04:02:19,883 - INFO - analytics-database
2025-11-27 04:02:19,883 - INFO - analytics-cassandra:active
2025-11-27 04:02:19,888 - INFO - analyticsdb-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:02:19,896 - INFO - query-engine:active
2025-11-27 04:02:19,896 - INFO - webui
2025-11-27 04:02:19,896 - INFO - webui:active
2025-11-27 04:02:19,896 - INFO - webui-middleware:active
2025-11-27 04:02:19,896 - INFO - redis:active
2025-11-27 04:02:19,896 - INFO - analytics_snmp
2025-11-27 04:02:19,901 - INFO - snmp-collector:active
2025-11-27 04:02:19,901 - INFO - snmp-topology:active
2025-11-27 04:02:19,901 - INFO - analytics_alarm
2025-11-27 04:02:19,901 - INFO - alarmgen:active
2025-11-27 04:02:19,988 - INFO - 10.0.0.23
2025-11-27 04:02:19,988 - INFO - config
2025-11-27 04:02:19,994 - INFO - config-nodemgr:active
2025-11-27 04:02:19,999 - INFO - api-server:active
2025-11-27 04:02:20,003 - INFO - schema:backup
2025-11-27 04:02:20,008 - INFO - svc-monitor:active
2025-11-27 04:02:20,012 - INFO - device-manager:active
2025-11-27 04:02:20,012 - INFO - config-database
2025-11-27 04:02:20,012 - INFO - config-cassandra:active
2025-11-27 04:02:20,012 - INFO - config-zookeeper:active
2025-11-27 04:02:20,012 - INFO - config-rabbitmq:active
2025-11-27 04:02:20,012 - INFO - control
2025-11-27 04:02:20,017 - INFO - control-nodemgr:active
2025-11-27 04:02:20,023 - INFO - control:active
2025-11-27 04:02:20,023 - INFO - named:active
2025-11-27 04:02:20,028 - INFO - dns:active
2025-11-27 04:02:20,028 - INFO - analytics
2025-11-27 04:02:20,033 - INFO - analytics-nodemgr:active
2025-11-27 04:02:20,038 - INFO - analytics-api:active
2025-11-27 04:02:20,044 - INFO - collector:active
2025-11-27 04:02:20,044 - INFO - analytics-database
2025-11-27 04:02:20,044 - INFO - analytics-cassandra:active
2025-11-27 04:02:20,049 - INFO - analyticsdb-nodemgr:active
2025-11-27 04:02:20,054 - INFO - query-engine:active
2025-11-27 04:02:20,054 - INFO - webui
2025-11-27 04:02:20,054 - INFO - webui:active
2025-11-27 04:02:20,054 - INFO - webui-middleware:active
2025-11-27 04:02:20,054 - INFO - redis:active
2025-11-27 04:02:20,054 - INFO - analytics_snmp
2025-11-27 04:02:20,059 - INFO - snmp-collector:active
2025-11-27 04:02:20,059 - INFO - snmp-topology:active
2025-11-27 04:02:20,059 - INFO - analytics_alarm
2025-11-27 04:02:20,059 - INFO - alarmgen:active
2025-11-27 04:02:20,149 - INFO - 10.0.0.19
2025-11-27 04:02:20,150 - INFO - vrouter
2025-11-27 04:02:20,155 - INFO - vrouter-nodemgr:active
2025-11-27 04:02:20,161 - INFO - agent:active
2025-11-27 04:02:20,247 - INFO - 10.0.0.130
2025-11-27 04:02:20,247 - INFO - vrouter
2025-11-27 04:02:20,253 - INFO - vrouter-nodemgr:active
2025-11-27 04:02:20,259 - INFO - agent:active
2025-11-27 04:02:20,259 - DEBUG - defaultdict(, {'10.0.0.33': {'config-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'control-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'analytics-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'analyticsdb-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}}})
2025-11-27 04:02:20,259 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 3
2025-11-27 04:02:30,371 - INFO - 10.0.0.27
2025-11-27 04:02:30,372 - INFO - config
2025-11-27 04:02:30,376 - INFO - config-nodemgr:active
2025-11-27 04:02:30,379 - INFO - api-server:active
2025-11-27 04:02:30,383 - INFO - schema:active
2025-11-27 04:02:30,386 - INFO - svc-monitor:backup
2025-11-27 04:02:30,388 - INFO - device-manager:backup
2025-11-27 04:02:30,388 - INFO - config-database
2025-11-27 04:02:30,388 - INFO - config-cassandra:active
2025-11-27 04:02:30,388 - INFO - config-zookeeper:active
2025-11-27 04:02:30,388 - INFO - config-rabbitmq:active
2025-11-27 04:02:30,388 - INFO - control
2025-11-27 04:02:30,392 - INFO - control-nodemgr:active
2025-11-27 04:02:30,396 - INFO - control:active
2025-11-27 04:02:30,397 - INFO - named:active
2025-11-27 04:02:30,401 - INFO - dns:active
2025-11-27 04:02:30,401 - INFO - analytics
2025-11-27 04:02:30,406 - INFO - analytics-nodemgr:active
2025-11-27 04:02:30,410 - INFO - analytics-api:active
2025-11-27 04:02:30,414 - INFO - collector:active
2025-11-27 04:02:30,414 - INFO - analytics-database
2025-11-27 04:02:30,414 - INFO - analytics-cassandra:active
2025-11-27 04:02:30,418 - INFO - analyticsdb-nodemgr:active
2025-11-27 04:02:30,423 - INFO - query-engine:active
2025-11-27 04:02:30,423 - INFO - webui
2025-11-27 04:02:30,424 - INFO - webui:active
2025-11-27 04:02:30,424 - INFO - webui-middleware:active
2025-11-27 04:02:30,424 - INFO - redis:active
2025-11-27 04:02:30,424 - INFO - analytics_snmp
2025-11-27 04:02:30,427 - INFO - snmp-collector:active
2025-11-27 04:02:30,427 - INFO - snmp-topology:active
2025-11-27 04:02:30,428 - INFO - analytics_alarm
2025-11-27 04:02:30,428 - INFO - alarmgen:active
2025-11-27 04:02:30,529 - INFO - 10.0.0.33
2025-11-27 04:02:30,529 - INFO - config
2025-11-27 04:02:30,535 - INFO - config-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:02:30,540 - INFO - api-server:active
2025-11-27 04:02:30,545 - INFO - schema:backup
2025-11-27 04:02:30,549 - INFO - svc-monitor:backup
2025-11-27 04:02:30,553 - INFO - device-manager:backup
2025-11-27 04:02:30,553 - INFO - config-database
2025-11-27 04:02:30,553 - INFO - config-cassandra:active
2025-11-27 04:02:30,553 - INFO - config-zookeeper:active
2025-11-27 04:02:30,553 - INFO - config-rabbitmq:active
2025-11-27 04:02:30,553 - INFO - control
2025-11-27 04:02:30,558 - INFO - control-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:02:30,564 - INFO - control:active
2025-11-27 04:02:30,565 - INFO - named:active
2025-11-27 04:02:30,569 - INFO - dns:active
2025-11-27 04:02:30,570 - INFO - analytics
2025-11-27 04:02:30,577 - INFO - analytics-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:02:30,581 - INFO - analytics-api:active
2025-11-27 04:02:30,586 - INFO - collector:active
2025-11-27 04:02:30,587 - INFO - analytics-database
2025-11-27 04:02:30,587 - INFO - analytics-cassandra:active
2025-11-27 04:02:30,593 - INFO - analyticsdb-nodemgr:initializing (NTP state unsynchronized. )
2025-11-27 04:02:30,598 - INFO - query-engine:active
2025-11-27 04:02:30,598 - INFO - webui
2025-11-27 04:02:30,598 - INFO - webui:active
2025-11-27 04:02:30,598 - INFO - webui-middleware:active
2025-11-27 04:02:30,598 - INFO - redis:active
2025-11-27 04:02:30,598 - INFO - analytics_snmp
2025-11-27 04:02:30,602 - INFO - snmp-collector:active
2025-11-27 04:02:30,602 - INFO - snmp-topology:active
2025-11-27 04:02:30,602 - INFO - analytics_alarm
2025-11-27 04:02:30,602 - INFO - alarmgen:active
2025-11-27 04:02:30,691 - INFO - 10.0.0.23
2025-11-27 04:02:30,691 - INFO - config
2025-11-27 04:02:30,696 - INFO - config-nodemgr:active
2025-11-27 04:02:30,700 - INFO - api-server:active
2025-11-27 04:02:30,703 - INFO - schema:backup
2025-11-27 04:02:30,708 - INFO - svc-monitor:active
2025-11-27 04:02:30,711 - INFO - device-manager:active
2025-11-27 04:02:30,711 - INFO - config-database
2025-11-27 04:02:30,711 - INFO - config-cassandra:active
2025-11-27 04:02:30,711 - INFO - config-zookeeper:active
2025-11-27 04:02:30,711 - INFO - config-rabbitmq:active
2025-11-27 04:02:30,712 - INFO - control
2025-11-27 04:02:30,716 - INFO - control-nodemgr:active
2025-11-27 04:02:30,721 - INFO - control:active
2025-11-27 04:02:30,721 - INFO - named:active
2025-11-27 04:02:30,726 - INFO - dns:active
2025-11-27 04:02:30,726 - INFO - analytics
2025-11-27 04:02:30,730 - INFO - analytics-nodemgr:active
2025-11-27 04:02:30,733 - INFO - analytics-api:active
2025-11-27 04:02:30,738 - INFO - collector:active
2025-11-27 04:02:30,738 - INFO - analytics-database
2025-11-27 04:02:30,738 - INFO - analytics-cassandra:active
2025-11-27 04:02:30,743 - INFO - analyticsdb-nodemgr:active
2025-11-27 04:02:30,747 - INFO - query-engine:active
2025-11-27 04:02:30,747 - INFO - webui
2025-11-27 04:02:30,747 - INFO - webui:active
2025-11-27 04:02:30,747 - INFO - webui-middleware:active
2025-11-27 04:02:30,748 - INFO - redis:active
2025-11-27 04:02:30,748 - INFO - analytics_snmp
2025-11-27 04:02:30,752 - INFO - snmp-collector:active
2025-11-27 04:02:30,752 - INFO - snmp-topology:active
2025-11-27 04:02:30,752 - INFO - analytics_alarm
2025-11-27 04:02:30,752 - INFO - alarmgen:active
2025-11-27 04:02:30,841 - INFO - 10.0.0.19
2025-11-27 04:02:30,842 - INFO - vrouter
2025-11-27 04:02:30,846 - INFO - vrouter-nodemgr:active
2025-11-27 04:02:30,851 - INFO - agent:active
2025-11-27 04:02:30,939 - INFO - 10.0.0.130
2025-11-27 04:02:30,939 - INFO - vrouter
2025-11-27 04:02:30,945 - INFO - vrouter-nodemgr:active
2025-11-27 04:02:30,951 - INFO - agent:active
2025-11-27 04:02:30,951 - DEBUG - defaultdict(, {'10.0.0.33': {'config-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'control-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'analytics-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'analyticsdb-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}}})
2025-11-27 04:02:30,951 - ERROR - Not all services up , Gave up!
2025-11-27 04:02:30,951 - INFO - Failed services are : defaultdict(, {'10.0.0.33': {'config-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'control-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'analytics-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}, 'analyticsdb-nodemgr': {'status': 'initializing', 'description': 'NTP state unsynchronized. '}}})
2025-11-27 04:02:30,951 - ERROR - contrail-status failed
2025-11-27 04:02:32,221 - DEBUG - Skipping xmpp flap check
2025-11-27 04:02:32,221 - INFO -
2025-11-27 04:02:32,221 - INFO - END TEST : test_contrail_status : FAILED[0:00:47]
2025-11-27 04:02:32,221 - INFO - --------------------------------------------------------------------------------