2026-06-10 16:26:52,540 - DEBUG - [10.0.0.50]: Running cmd : hostname
2026-06-10 16:26:52,740 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-5926-1
2026-06-10 16:26:52,741 - DEBUG - [10.0.0.50]: Running cmd : hostname -f
2026-06-10 16:26:52,800 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-5926-1.
2026-06-10 16:26:52,801 - DEBUG - [10.0.0.50]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2026-06-10 16:26:52,911 - DEBUG - Output : NAMES
contrail_test_56iB4Lho9
vrouter_vrouter-agent_1
vrouter_nodemgr_1
rsyslogd_rsyslogd_1
analytics_snmp_nodemgr_1
analytics_snmp_snmp-collector_1
analytics_snmp_topology_1
analytics_alarm_nodemgr_1
analytics_alarm_alarm-gen_1
analytics_alarm_kafka_1
analytics_collector_1
analytics_api_1
analytics_nodemgr_1
analytics_database_nodemgr_1
analytics_database_query-engine_1
analytics_database_cassandra_1
control_named_1
control_nodemgr_1
control_dns_1
control_control_1
webui_web_1
webui_job_1
config_devicemgr_1
config_dnsmasq_1
config_nodemgr_1
config_api_1
config_svcmonitor_1
config_schema_1
config_database_nodemgr_1
config_database_rabbitmq_1
config_database_cassandra_1
config_database_zookeeper_1
redis_redis_1
barbican_worker
barbican_keystone_listener
barbican_api
horizon
heat_api_cfn
heat_api
nova_compute
nova_libvirt
nova_ssh
nova_novncproxy
nova_conductor
nova_api
nova_scheduler
glance_api
keystone
keystone_fernet
keystone_ssh
rabbitmq
memcached
mariadb
mariadb_clustercheck
cron
kolla_toolbox
fluentd
2026-06-10 16:26:52,912 - DEBUG - [10.0.0.50]: Running cmd : getent hosts 10.0.0.50 | head -n 1 | awk '{print $2}'
2026-06-10 16:26:52,969 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-5926-1.
2026-06-10 16:26:52,969 - DEBUG - [10.0.0.50]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2026-06-10 16:26:53,036 - DEBUG - Output : 127.0.0.1/8
10.0.0.50/24
172.17.0.1/16
10.20.0.17/24
2026-06-10 16:26:53,036 - DEBUG - [10.0.0.50]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2026-06-10 16:26:53,100 - DEBUG - Output : 10.20.0.17/24
2026-06-10 16:26:53,101 - DEBUG - [10.0.0.50]: Running cmd : getent hosts 10.20.0.17 | head -n 1 | awk '{print $2}'
2026-06-10 16:26:53,164 - DEBUG - Output :
2026-06-10 16:26:54,892 - INFO - Using existing project ['default-domain', 'admin'](399c1c70-c2e2-460a-b1e6-09923f923677)
2026-06-10 16:26:55,036 - INFO - ================================================================================
2026-06-10 16:26:55,036 - INFO - STARTING TEST : test_contrail_status
2026-06-10 16:26:55,036 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active
2026-06-10 16:26:55,164 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:26:55,170 - DEBUG - Nothing to compare xmpp stats {'10.0.0.50': {'10.20.0.17': '0'}} with
2026-06-10 16:26:55,170 - INFO - Initial checks done. Running the testcase now
2026-06-10 16:26:55,170 - INFO -
2026-06-10 16:26:55,170 - DEBUG - [10.0.0.50]: Running cmd : docker ps -f status=running --format {{.Names}} 2>/dev/null
2026-06-10 16:26:55,261 - DEBUG - Output : contrail_test_56iB4Lho9
vrouter_vrouter-agent_1
vrouter_provisioner_1
vrouter_nodemgr_1
rsyslogd_rsyslogd_1
analytics_snmp_nodemgr_1
analytics_snmp_snmp-collector_1
analytics_snmp_provisioner_1
analytics_snmp_topology_1
analytics_alarm_nodemgr_1
analytics_alarm_provisioner_1
analytics_alarm_alarm-gen_1
analytics_alarm_kafka_1
analytics_collector_1
analytics_provisioner_1
analytics_api_1
analytics_nodemgr_1
analytics_database_provisioner_1
analytics_database_nodemgr_1
analytics_database_query-engine_1
analytics_database_cassandra_1
control_named_1
control_nodemgr_1
control_dns_1
control_provisioner_1
control_control_1
webui_web_1
webui_job_1
config_devicemgr_1
config_dnsmasq_1
config_nodemgr_1
config_provisioner_1
config_api_1
config_svcmonitor_1
config_schema_1
config_database_provisioner_1
config_database_nodemgr_1
config_database_rabbitmq_1
config_database_cassandra_1
config_database_zookeeper_1
redis_redis_1
barbican_worker
barbican_keystone_listener
barbican_api
horizon
heat_api_cfn
heat_api
nova_compute
nova_libvirt
nova_ssh
nova_novncproxy
nova_conductor
nova_api
nova_scheduler
placement_api
glance_api
keystone
keystone_fernet
keystone_ssh
rabbitmq
memcached
mariadb
mariadb_clustercheck
cron
kolla_toolbox
fluentd
2026-06-10 16:26:55,262 - INFO - 10.0.0.50
2026-06-10 16:26:55,262 - INFO - config
2026-06-10 16:26:55,267 - INFO - config-nodemgr:active
2026-06-10 16:26:55,270 - INFO - api-server:active
2026-06-10 16:26:55,274 - INFO - schema:active
2026-06-10 16:26:55,277 - INFO - svc-monitor:active
2026-06-10 16:26:55,280 - INFO - device-manager:active
2026-06-10 16:26:55,280 - INFO - config-database
2026-06-10 16:26:55,281 - INFO - config-cassandra:active
2026-06-10 16:26:55,281 - INFO - config-zookeeper:active
2026-06-10 16:26:55,281 - INFO - config-rabbitmq:active
2026-06-10 16:26:55,281 - INFO - vrouter
2026-06-10 16:26:55,285 - INFO - vrouter-nodemgr:active
2026-06-10 16:26:55,290 - INFO - agent:active
2026-06-10 16:26:55,290 - INFO - control
2026-06-10 16:26:55,295 - INFO - control-nodemgr:active
2026-06-10 16:26:55,299 - INFO - control:active
2026-06-10 16:26:55,300 - INFO - named:active
2026-06-10 16:26:55,305 - INFO - dns:active
2026-06-10 16:26:55,305 - INFO - analytics
2026-06-10 16:26:55,310 - INFO - analytics-nodemgr:active
2026-06-10 16:26:55,314 - INFO - analytics-api:active
2026-06-10 16:26:55,319 - INFO - collector:active
2026-06-10 16:26:55,319 - INFO - analytics-database
2026-06-10 16:26:55,319 - INFO - analytics-cassandra:active
2026-06-10 16:26:55,323 - INFO - analyticsdb-nodemgr:active
2026-06-10 16:26:55,327 - INFO - query-engine:active
2026-06-10 16:26:55,327 - INFO - webui
2026-06-10 16:26:55,327 - INFO - webui:active
2026-06-10 16:26:55,327 - INFO - webui-middleware:active
2026-06-10 16:26:55,327 - INFO - redis:active
2026-06-10 16:26:55,327 - INFO - analytics_snmp
2026-06-10 16:26:55,330 - INFO - snmp-collector:active
2026-06-10 16:26:55,330 - INFO - snmp-topology:active
2026-06-10 16:26:55,330 - INFO - analytics_alarm
2026-06-10 16:26:55,330 - INFO - alarmgen:active
2026-06-10 16:26:55,330 - INFO - All the contrail services are up on all nodes
2026-06-10 16:26:55,330 - INFO - contrail-status passed
2026-06-10 16:26:55,446 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:26:55,452 - DEBUG - No XMPP flaps were noticed during the test
2026-06-10 16:26:55,452 - INFO - END TEST : test_contrail_status : PASSED[0:00:00]
2026-06-10 16:26:55,452 - INFO - --------------------------------------------------------------------------------
2026-06-10 16:26:58,126 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:26:58,731 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:26:59,777 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:26:59,784 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it config_api_1 /bin/bash -c 'db-manage check'
2026-06-10 16:27:03,053 - DEBUG - Output : 2026-06-10 16:27:02,310 INFO: (v1.41) Checker check_zk_mode_and_node_count: Success
2026-06-10 16:27:02,333 INFO: (v1.41) Checker check_cassandra_keyspace_replication: Success
2026-06-10 16:27:02,415 INFO: (v1.41) Checker check_obj_mandatory_fields: Success
2026-06-10 16:27:02,595 INFO: (v1.41) Checker check_orphan_resources: Success
2026-06-10 16:27:02,778 INFO: (v1.41) Checker check_fq_name_uuid_match: Success
2026-06-10 16:27:02,778 WARNING: Be careful, that check can return false positive errors if stale FQ names and stale resources were not cleaned before. Run at least commands 'clean_obj_missing_mandatory_fields', 'clean_orphan_resources' and 'clean_stale_fq_names' before.
2026-06-10 16:27:02,870 INFO: (v1.41) Checker check_duplicate_fq_name: Success
2026-06-10 16:27:02,882 INFO: (v1.41) Checker check_route_targets_routing_instance_backrefs: Success
2026-06-10 16:27:02,888 INFO: (v1.41) Checker check_subnet_uuid: Success
2026-06-10 16:27:02,896 INFO: (v1.41) Checker check_subnet_addr_alloc: Success
2026-06-10 16:27:02,922 INFO: (v1.41) Checker check_route_targets_id: Success
2026-06-10 16:27:02,931 INFO: (v1.41) Checker check_virtual_networks_id: Success
2026-06-10 16:27:02,937 INFO: (v1.41) Checker check_security_groups_id: Success
2026-06-10 16:27:02,940 INFO: (v1.41) Checker check_ipam_subnet_method: Success
2026-06-10 16:27:02,942 INFO: (v1.41) Checker check_aggregated_ethernet_id: Success
2026-06-10 16:27:03,210 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:27:03,394 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:27:03,566 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:27:03,738 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:27:03,891 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:27:06,485 - INFO - admin user already present
2026-06-10 16:27:07,274 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:27:07,928 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:27:11,193 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:11,414 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:17,603 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:18,211 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:18,370 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:18,598 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:21,678 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:22,263 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:25,406 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:26,871 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:30,051 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:30,632 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:33,797 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:33,965 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:34,895 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:35,061 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:38,070 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:38,752 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:42,082 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:42,904 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:46,038 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:46,615 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:49,692 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:51,848 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:54,970 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:55,535 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:58,571 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:59,183 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:59,338 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:28:59,890 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:29:00,060 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:29:00,639 - DEBUG - Requesting: http://10.0.0.50:8085/Snh_AgentXmppConnectionStatusReq?
2026-06-10 16:32:50,363 - INFO - Using existing project ['default-domain', 'admin'](399c1c70-c2e2-460a-b1e6-09923f923677)
2026-06-10 16:32:50,372 - INFO - ================================================================================
2026-06-10 16:32:50,372 - INFO - STARTING TEST : test_contrail_status
2026-06-10 16:32:50,372 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active
2026-06-10 16:32:50,495 - DEBUG - Skipping xmpp flap check
2026-06-10 16:32:50,495 - INFO - Initial checks done. Running the testcase now
2026-06-10 16:32:50,495 - INFO -
2026-06-10 16:32:50,598 - INFO - 10.0.0.50
2026-06-10 16:32:50,598 - INFO - config
2026-06-10 16:32:50,604 - INFO - config-nodemgr:active
2026-06-10 16:32:50,608 - INFO - api-server:initializing (Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-06-10 16:32:41.564541] connection down)
2026-06-10 16:32:50,612 - INFO - schema:active
2026-06-10 16:32:50,616 - INFO - svc-monitor:active
2026-06-10 16:32:50,620 - INFO - device-manager:active
2026-06-10 16:32:50,621 - INFO - config-database
2026-06-10 16:32:50,621 - INFO - config-cassandra:active
2026-06-10 16:32:50,621 - INFO - config-zookeeper:active
2026-06-10 16:32:50,621 - INFO - config-rabbitmq:active
2026-06-10 16:32:50,621 - INFO - vrouter
2026-06-10 16:32:50,626 - INFO - vrouter-nodemgr:active
2026-06-10 16:32:50,632 - INFO - agent:active
2026-06-10 16:32:50,632 - INFO - control
2026-06-10 16:32:50,638 - INFO - control-nodemgr:active
2026-06-10 16:32:50,642 - INFO - control:active
2026-06-10 16:32:50,642 - INFO - named:active
2026-06-10 16:32:50,646 - INFO - dns:active
2026-06-10 16:32:50,647 - INFO - analytics
2026-06-10 16:32:50,651 - INFO - analytics-nodemgr:active
2026-06-10 16:32:50,655 - INFO - analytics-api:active
2026-06-10 16:32:50,660 - INFO - collector:active
2026-06-10 16:32:50,660 - INFO - analytics-database
2026-06-10 16:32:50,660 - INFO - analytics-cassandra:active
2026-06-10 16:32:50,665 - INFO - analyticsdb-nodemgr:active
2026-06-10 16:32:50,669 - INFO - query-engine:active
2026-06-10 16:32:50,669 - INFO - webui
2026-06-10 16:32:50,670 - INFO - webui:active
2026-06-10 16:32:50,670 - INFO - webui-middleware:active
2026-06-10 16:32:50,670 - INFO - redis:active
2026-06-10 16:32:50,670 - INFO - analytics_snmp
2026-06-10 16:32:50,673 - INFO - snmp-collector:active
2026-06-10 16:32:50,674 - INFO - snmp-topology:active
2026-06-10 16:32:50,674 - INFO - analytics_alarm
2026-06-10 16:32:50,674 - INFO - alarmgen:active
2026-06-10 16:32:50,674 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-06-10 16:32:41.564541] connection down'}}})
2026-06-10 16:32:50,674 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 0
2026-06-10 16:33:00,740 - INFO - 10.0.0.50
2026-06-10 16:33:00,740 - INFO - config
2026-06-10 16:33:00,746 - INFO - config-nodemgr:active
2026-06-10 16:33:00,751 - INFO - api-server:initializing (Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-06-10 16:32:41.564541] connection down)
2026-06-10 16:33:00,756 - INFO - schema:active
2026-06-10 16:33:00,760 - INFO - svc-monitor:active
2026-06-10 16:33:00,765 - INFO - device-manager:active
2026-06-10 16:33:00,765 - INFO - config-database
2026-06-10 16:33:00,765 - INFO - config-cassandra:active
2026-06-10 16:33:00,765 - INFO - config-zookeeper:active
2026-06-10 16:33:00,765 - INFO - config-rabbitmq:active
2026-06-10 16:33:00,765 - INFO - vrouter
2026-06-10 16:33:00,770 - INFO - vrouter-nodemgr:active
2026-06-10 16:33:00,775 - INFO - agent:active
2026-06-10 16:33:00,775 - INFO - control
2026-06-10 16:33:00,780 - INFO - control-nodemgr:active
2026-06-10 16:33:00,786 - INFO - control:active
2026-06-10 16:33:00,786 - INFO - named:active
2026-06-10 16:33:00,791 - INFO - dns:active
2026-06-10 16:33:00,792 - INFO - analytics
2026-06-10 16:33:00,797 - INFO - analytics-nodemgr:active
2026-06-10 16:33:00,802 - INFO - analytics-api:active
2026-06-10 16:33:00,807 - INFO - collector:active
2026-06-10 16:33:00,807 - INFO - analytics-database
2026-06-10 16:33:00,808 - INFO - analytics-cassandra:active
2026-06-10 16:33:00,812 - INFO - analyticsdb-nodemgr:active
2026-06-10 16:33:00,818 - INFO - query-engine:active
2026-06-10 16:33:00,818 - INFO - webui
2026-06-10 16:33:00,818 - INFO - webui:active
2026-06-10 16:33:00,818 - INFO - webui-middleware:active
2026-06-10 16:33:00,818 - INFO - redis:active
2026-06-10 16:33:00,818 - INFO - analytics_snmp
2026-06-10 16:33:00,822 - INFO - snmp-collector:active
2026-06-10 16:33:00,823 - INFO - snmp-topology:active
2026-06-10 16:33:00,823 - INFO - analytics_alarm
2026-06-10 16:33:00,823 - INFO - alarmgen:active
2026-06-10 16:33:00,823 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-06-10 16:32:41.564541] connection down'}}})
2026-06-10 16:33:00,823 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 1
2026-06-10 16:33:10,891 - INFO - 10.0.0.50
2026-06-10 16:33:10,891 - INFO - config
2026-06-10 16:33:10,896 - INFO - config-nodemgr:active
2026-06-10 16:33:10,900 - INFO - api-server:initializing (Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-06-10 16:32:41.564541] connection down)
2026-06-10 16:33:10,903 - INFO - schema:active
2026-06-10 16:33:10,907 - INFO - svc-monitor:active
2026-06-10 16:33:10,911 - INFO - device-manager:active
2026-06-10 16:33:10,911 - INFO - config-database
2026-06-10 16:33:10,911 - INFO - config-cassandra:active
2026-06-10 16:33:10,911 - INFO - config-zookeeper:active
2026-06-10 16:33:10,911 - INFO - config-rabbitmq:active
2026-06-10 16:33:10,911 - INFO - vrouter
2026-06-10 16:33:10,915 - INFO - vrouter-nodemgr:active
2026-06-10 16:33:10,919 - INFO - agent:active
2026-06-10 16:33:10,919 - INFO - control
2026-06-10 16:33:10,923 - INFO - control-nodemgr:active
2026-06-10 16:33:10,927 - INFO - control:active
2026-06-10 16:33:10,927 - INFO - named:active
2026-06-10 16:33:10,931 - INFO - dns:active
2026-06-10 16:33:10,931 - INFO - analytics
2026-06-10 16:33:10,935 - INFO - analytics-nodemgr:active
2026-06-10 16:33:10,938 - INFO - analytics-api:active
2026-06-10 16:33:10,942 - INFO - collector:active
2026-06-10 16:33:10,942 - INFO - analytics-database
2026-06-10 16:33:10,942 - INFO - analytics-cassandra:active
2026-06-10 16:33:10,946 - INFO - analyticsdb-nodemgr:active
2026-06-10 16:33:10,951 - INFO - query-engine:active
2026-06-10 16:33:10,951 - INFO - webui
2026-06-10 16:33:10,951 - INFO - webui:active
2026-06-10 16:33:10,951 - INFO - webui-middleware:active
2026-06-10 16:33:10,951 - INFO - redis:active
2026-06-10 16:33:10,951 - INFO - analytics_snmp
2026-06-10 16:33:10,954 - INFO - snmp-collector:active
2026-06-10 16:33:10,954 - INFO - snmp-topology:active
2026-06-10 16:33:10,954 - INFO - analytics_alarm
2026-06-10 16:33:10,954 - INFO - alarmgen:active
2026-06-10 16:33:10,954 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-06-10 16:32:41.564541] connection down'}}})
2026-06-10 16:33:10,954 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 2
2026-06-10 16:33:21,021 - INFO - 10.0.0.50
2026-06-10 16:33:21,021 - INFO - config
2026-06-10 16:33:21,026 - INFO - config-nodemgr:active
2026-06-10 16:33:21,030 - INFO - api-server:initializing (Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-06-10 16:32:41.564541] connection down)
2026-06-10 16:33:21,034 - INFO - schema:active
2026-06-10 16:33:21,037 - INFO - svc-monitor:active
2026-06-10 16:33:21,040 - INFO - device-manager:active
2026-06-10 16:33:21,040 - INFO - config-database
2026-06-10 16:33:21,040 - INFO - config-cassandra:active
2026-06-10 16:33:21,040 - INFO - config-zookeeper:active
2026-06-10 16:33:21,040 - INFO - config-rabbitmq:active
2026-06-10 16:33:21,040 - INFO - vrouter
2026-06-10 16:33:21,044 - INFO - vrouter-nodemgr:active
2026-06-10 16:33:21,048 - INFO - agent:active
2026-06-10 16:33:21,048 - INFO - control
2026-06-10 16:33:21,051 - INFO - control-nodemgr:active
2026-06-10 16:33:21,055 - INFO - control:active
2026-06-10 16:33:21,055 - INFO - named:active
2026-06-10 16:33:21,059 - INFO - dns:active
2026-06-10 16:33:21,059 - INFO - analytics
2026-06-10 16:33:21,063 - INFO - analytics-nodemgr:active
2026-06-10 16:33:21,066 - INFO - analytics-api:active
2026-06-10 16:33:21,070 - INFO - collector:active
2026-06-10 16:33:21,070 - INFO - analytics-database
2026-06-10 16:33:21,070 - INFO - analytics-cassandra:active
2026-06-10 16:33:21,075 - INFO - analyticsdb-nodemgr:active
2026-06-10 16:33:21,080 - INFO - query-engine:active
2026-06-10 16:33:21,080 - INFO - webui
2026-06-10 16:33:21,080 - INFO - webui:active
2026-06-10 16:33:21,080 - INFO - webui-middleware:active
2026-06-10 16:33:21,080 - INFO - redis:active
2026-06-10 16:33:21,080 - INFO - analytics_snmp
2026-06-10 16:33:21,084 - INFO - snmp-collector:active
2026-06-10 16:33:21,084 - INFO - snmp-topology:active
2026-06-10 16:33:21,084 - INFO - analytics_alarm
2026-06-10 16:33:21,084 - INFO - alarmgen:active
2026-06-10 16:33:21,084 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-06-10 16:32:41.564541] connection down'}}})
2026-06-10 16:33:21,084 - DEBUG - Not all services up. Sleeping for 10 seconds. iteration: 3
2026-06-10 16:33:31,141 - INFO - 10.0.0.50
2026-06-10 16:33:31,141 - INFO - config
2026-06-10 16:33:31,147 - INFO - config-nodemgr:active
2026-06-10 16:33:31,151 - INFO - api-server:initializing (Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-06-10 16:32:41.564541] connection down)
2026-06-10 16:33:31,155 - INFO - schema:active
2026-06-10 16:33:31,159 - INFO - svc-monitor:active
2026-06-10 16:33:31,163 - INFO - device-manager:active
2026-06-10 16:33:31,163 - INFO - config-database
2026-06-10 16:33:31,163 - INFO - config-cassandra:active
2026-06-10 16:33:31,163 - INFO - config-zookeeper:active
2026-06-10 16:33:31,163 - INFO - config-rabbitmq:active
2026-06-10 16:33:31,163 - INFO - vrouter
2026-06-10 16:33:31,168 - INFO - vrouter-nodemgr:active
2026-06-10 16:33:31,173 - INFO - agent:active
2026-06-10 16:33:31,173 - INFO - control
2026-06-10 16:33:31,178 - INFO - control-nodemgr:active
2026-06-10 16:33:31,184 - INFO - control:active
2026-06-10 16:33:31,184 - INFO - named:active
2026-06-10 16:33:31,189 - INFO - dns:active
2026-06-10 16:33:31,189 - INFO - analytics
2026-06-10 16:33:31,195 - INFO - analytics-nodemgr:active
2026-06-10 16:33:31,199 - INFO - analytics-api:active
2026-06-10 16:33:31,204 - INFO - collector:active
2026-06-10 16:33:31,205 - INFO - analytics-database
2026-06-10 16:33:31,205 - INFO - analytics-cassandra:active
2026-06-10 16:33:31,210 - INFO - analyticsdb-nodemgr:active
2026-06-10 16:33:31,214 - INFO - query-engine:active
2026-06-10 16:33:31,214 - INFO - webui
2026-06-10 16:33:31,215 - INFO - webui:active
2026-06-10 16:33:31,215 - INFO - webui-middleware:active
2026-06-10 16:33:31,215 - INFO - redis:active
2026-06-10 16:33:31,215 - INFO - analytics_snmp
2026-06-10 16:33:31,219 - INFO - snmp-collector:active
2026-06-10 16:33:31,219 - INFO - snmp-topology:active
2026-06-10 16:33:31,219 - INFO - analytics_alarm
2026-06-10 16:33:31,219 - INFO - alarmgen:active
2026-06-10 16:33:31,219 - DEBUG - defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-06-10 16:32:41.564541] connection down'}}})
2026-06-10 16:33:31,219 - ERROR - Not all services up , Gave up!
2026-06-10 16:33:31,219 - INFO - Failed services are : defaultdict(, {'10.0.0.50': {'api-server': {'status': 'initializing', 'description': 'Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.50:5000/v3/domains? at UTC 2026-06-10 16:32:41.564541] connection down'}}})
2026-06-10 16:33:31,219 - ERROR - contrail-status failed
2026-06-10 16:33:31,342 - DEBUG - Skipping xmpp flap check
2026-06-10 16:33:31,342 - INFO -
2026-06-10 16:33:31,342 - INFO - END TEST : test_contrail_status : FAILED[0:00:41]
2026-06-10 16:33:31,343 - INFO - --------------------------------------------------------------------------------