2025-08-11 10:52:56,011 - INFO - Using existing project ['default-domain', 'admin'](74e57a64-053a-4ab9-9a41-98124cb3c978)
2025-08-11 10:52:56,025 - INFO - ================================================================================
2025-08-11 10:52:56,025 - INFO - STARTING TEST : test_db_nodemgr_status
2025-08-11 10:52:56,025 - INFO - TEST DESCRIPTION : Test to verify db nodemgr status
2025-08-11 10:52:56,622 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with
2025-08-11 10:52:56,622 - INFO - Initial checks done. Running the testcase now
2025-08-11 10:52:56,623 - INFO -
2025-08-11 10:52:56,623 - DEBUG - Verifying if db node_mgr running...
2025-08-11 10:52:56,623 - INFO - Verifying through collector 10.0.0.22 for db node cn-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 10:52:56,623 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/database-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 10:52:56,913 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/database-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 10:52:56,974 - INFO - Verifying through collector 10.0.0.22 for db node cn-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 10:52:56,974 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/database-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat
2025-08-11 10:52:57,031 - INFO - Verifying through collector 10.0.0.22 for db node cn-jenkins-deploy-platform-ansible-os-3803-3.
2025-08-11 10:52:57,032 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/database-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat
2025-08-11 10:52:57,691 - DEBUG - No XMPP flaps were noticed during the test
2025-08-11 10:52:57,691 - INFO - END TEST : test_db_nodemgr_status : PASSED[0:00:01]
2025-08-11 10:52:57,691 - INFO - --------------------------------------------------------------------------------
2025-08-11 10:52:57,693 - INFO - ================================================================================
2025-08-11 10:52:57,693 - INFO - STARTING TEST : test_verify_generator_collector_connections
2025-08-11 10:52:57,693 - INFO - TEST DESCRIPTION :
Description: Verify generator:module connections to collector
1.Verify all generators connected to collector - fails otherwise
2.Get the xmpp peers in vrouter uve and get the active xmpp peer out of it
3.Verify from agent introspect that active xmpp matches with step 2 - fails otherwise
4.Get bgp peers from bgp-peer uve and verify from control node introspect that that matches - fails otherwise
Maintainer: sandipd@juniper.net
2025-08-11 10:52:58,310 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with
2025-08-11 10:52:58,310 - INFO - Initial checks done. Running the testcase now
2025-08-11 10:52:58,310 - INFO -
2025-08-11 10:52:58,318 - INFO - IP 10.0.0.56 port 8083 connected to collector 10.0.0.22
2025-08-11 10:52:58,324 - INFO - IP 10.0.0.38 port 8083 connected to collector 10.0.0.22
2025-08-11 10:52:58,330 - INFO - IP 10.0.0.22 port 8083 connected to collector 10.0.0.22
2025-08-11 10:52:58,337 - INFO - IP 10.0.0.56 port 8084 connected to collector 10.0.0.22
2025-08-11 10:52:58,345 - INFO - IP 10.0.0.38 port 8084 connected to collector 10.0.0.22
2025-08-11 10:52:58,361 - INFO - IP 10.0.0.22 port 8084 connected to collector 10.0.0.38
2025-08-11 10:52:58,367 - INFO - IP 10.0.0.56 port 8087 connected to collector 10.0.0.38
2025-08-11 10:52:58,371 - INFO - IP 10.0.0.56 port 8088 connected to collector 10.0.0.56
2025-08-11 10:52:58,376 - INFO - IP 10.0.0.56 port 8090 connected to collector 10.0.0.56
2025-08-11 10:52:58,380 - INFO - IP 10.0.0.38 port 8090 connected to collector 10.0.0.56
2025-08-11 10:52:58,384 - INFO - IP 10.0.0.22 port 8090 connected to collector 10.0.0.22
2025-08-11 10:52:58,390 - INFO - IP 10.0.0.56 port 8091 connected to collector 10.0.0.56
2025-08-11 10:52:58,396 - INFO - IP 10.0.0.38 port 8091 connected to collector 10.0.0.56
2025-08-11 10:52:58,402 - INFO - IP 10.0.0.22 port 8091 connected to collector 10.0.0.38
2025-08-11 10:52:58,402 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:58,403 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-control:0?flat
2025-08-11 10:52:58,699 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-control:0?flat
2025-08-11 10:52:58,789 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-control:0 is connected to collector 10.0.0.56
2025-08-11 10:52:58,789 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:58,789 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-dns:0?flat
2025-08-11 10:52:58,869 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-dns:0 is connected to collector 10.0.0.56
2025-08-11 10:52:58,869 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:58,869 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-control:0?flat
2025-08-11 10:52:58,955 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-control:0 is connected to collector 10.0.0.56
2025-08-11 10:52:58,955 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:58,955 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-dns:0?flat
2025-08-11 10:52:58,984 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-dns:0 is connected to collector 10.0.0.56
2025-08-11 10:52:58,984 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:58,984 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-control:0?flat
2025-08-11 10:52:59,031 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-control:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,031 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,031 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-dns:0?flat
2025-08-11 10:52:59,058 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-dns:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,058 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,058 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-agent:0?flat
2025-08-11 10:52:59,095 - INFO - Validated that an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-agent:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,095 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,095 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-agent:0?flat
2025-08-11 10:52:59,132 - INFO - Validated that an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-agent:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,132 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,132 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-schema:0?flat
2025-08-11 10:52:59,151 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-schema:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,151 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,151 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-svc-monitor:0?flat
2025-08-11 10:52:59,169 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-svc-monitor:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,169 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generators
2025-08-11 10:52:59,184 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,184 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-api:0?flat
2025-08-11 10:52:59,201 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-api:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,201 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,201 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Config:contrail-api:0?flat
2025-08-11 10:52:59,218 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Config:contrail-api:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,218 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,218 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Config:contrail-api:0?flat
2025-08-11 10:52:59,235 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Config:contrail-api:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,235 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generators
2025-08-11 10:52:59,249 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,249 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0?flat
2025-08-11 10:52:59,266 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,266 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,266 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0?flat
2025-08-11 10:52:59,284 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,284 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,284 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0?flat
2025-08-11 10:52:59,301 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,301 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,301 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0?flat
2025-08-11 10:52:59,331 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,331 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,331 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0?flat
2025-08-11 10:52:59,359 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,359 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,359 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0?flat
2025-08-11 10:52:59,376 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,376 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,376 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0?flat
2025-08-11 10:52:59,393 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,393 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,393 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0?flat
2025-08-11 10:52:59,410 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,410 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,410 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0?flat
2025-08-11 10:52:59,426 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,426 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,426 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-collector:0?flat
2025-08-11 10:52:59,455 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-collector:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,455 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,455 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Database:contrail-query-engine:0?flat
2025-08-11 10:52:59,480 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Database:contrail-query-engine:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,481 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,481 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-collector:0?flat
2025-08-11 10:52:59,509 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-collector:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,509 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,509 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Database:contrail-query-engine:0?flat
2025-08-11 10:52:59,535 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Database:contrail-query-engine:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,535 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,535 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-collector:0?flat
2025-08-11 10:52:59,561 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-collector:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,561 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:52:59,561 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Database:contrail-query-engine:0?flat
2025-08-11 10:52:59,586 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Database:contrail-query-engine:0 is connected to collector 10.0.0.56
2025-08-11 10:52:59,586 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-agent:0?flat
2025-08-11 10:52:59,620 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3803-1?flat
2025-08-11 10:52:59,654 - INFO - Active xmpp peer in an-jenkins-deploy-platform-ansible-os-3803-1 vrouter uve is 10.20.0.25
2025-08-11 10:52:59,678 - DEBUG - Requesting: http://10.20.0.193:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 10:52:59,689 - INFO - Active xmpp peer in an-jenkins-deploy-platform-ansible-os-3803-1 agent introspect is 10.20.0.25
2025-08-11 10:52:59,689 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-agent:0?flat
2025-08-11 10:52:59,725 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3803-2?flat
2025-08-11 10:52:59,759 - INFO - Active xmpp peer in an-jenkins-deploy-platform-ansible-os-3803-2 vrouter uve is 10.20.0.254
2025-08-11 10:52:59,783 - DEBUG - Requesting: http://10.20.0.129:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 10:52:59,798 - INFO - Active xmpp peer in an-jenkins-deploy-platform-ansible-os-3803-2 agent introspect is 10.20.0.254
2025-08-11 10:52:59,798 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-agent:0?flat
2025-08-11 10:52:59,835 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3803-1?flat
2025-08-11 10:52:59,868 - INFO - Vrouter an-jenkins-deploy-platform-ansible-os-3803-1 connected to {'10.20.0.25', '10.20.0.17'} xmpp peers out of {'10.20.0.25', '10.20.0.17', '10.20.0.254'}
2025-08-11 10:52:59,868 - INFO - xmpp peer correctly displayed as ['10.20.0.17', '10.20.0.25'] for vrouter an-jenkins-deploy-platform-ansible-os-3803-1
2025-08-11 10:52:59,868 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-agent:0?flat
2025-08-11 10:52:59,902 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3803-2?flat
2025-08-11 10:52:59,934 - INFO - Vrouter an-jenkins-deploy-platform-ansible-os-3803-2 connected to {'10.20.0.25', '10.20.0.254'} xmpp peers out of {'10.20.0.25', '10.20.0.17', '10.20.0.254'}
2025-08-11 10:52:59,934 - INFO - xmpp peer correctly displayed as ['10.20.0.254', '10.20.0.25'] for vrouter an-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 10:52:59,934 - INFO - Verifying through opserver in 10.0.0.22
2025-08-11 10:52:59,935 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-1?flat
2025-08-11 10:52:59,956 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-2?flat
2025-08-11 10:52:59,975 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-3?flat
2025-08-11 10:52:59,995 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-1?flat
2025-08-11 10:53:00,014 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-2?flat
2025-08-11 10:53:00,034 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-3?flat
2025-08-11 10:53:00,052 - DEBUG - Verifying for cn-jenkins-deploy-platform-ansible-os-3803-1 bgp-router uve
2025-08-11 10:53:00,052 - INFO - Bgp peers = 2
2025-08-11 10:53:00,052 - DEBUG - Verifying for cn-jenkins-deploy-platform-ansible-os-3803-2 bgp-router uve
2025-08-11 10:53:00,052 - INFO - Bgp peers = 2
2025-08-11 10:53:00,053 - DEBUG - Verifying for cn-jenkins-deploy-platform-ansible-os-3803-3 bgp-router uve
2025-08-11 10:53:00,053 - INFO - Bgp peers = 2
2025-08-11 10:53:00,053 - INFO - Total xmpp peer connections found = 4
2025-08-11 10:53:00,670 - DEBUG - No XMPP flaps were noticed during the test
2025-08-11 10:53:00,670 - INFO - END TEST : test_verify_generator_collector_connections : PASSED[0:00:03]
2025-08-11 10:53:00,670 - INFO - --------------------------------------------------------------------------------
2025-08-11 10:53:00,673 - INFO - ================================================================================
2025-08-11 10:53:00,673 - INFO - STARTING TEST : test_verify_generator_connections_to_collector_node
2025-08-11 10:53:00,673 - INFO - TEST DESCRIPTION : Test to validate generator connections
2025-08-11 10:53:01,282 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with
2025-08-11 10:53:01,282 - INFO - Initial checks done. Running the testcase now
2025-08-11 10:53:01,282 - INFO -
2025-08-11 10:53:01,282 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,283 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-agent:0?flat
2025-08-11 10:53:01,326 - INFO - Validated that an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-agent:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,326 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,326 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-nodemgr:0?flat
2025-08-11 10:53:01,349 - INFO - Validated that an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-nodemgr:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,349 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,349 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-agent:0?flat
2025-08-11 10:53:01,392 - INFO - Validated that an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-agent:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,392 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,392 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-nodemgr:0?flat
2025-08-11 10:53:01,411 - INFO - Validated that an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-nodemgr:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,411 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,411 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-nodemgr:0?flat
2025-08-11 10:53:01,443 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-nodemgr:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,443 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,443 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-collector:0?flat
2025-08-11 10:53:01,481 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-collector:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,482 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,482 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0?flat
2025-08-11 10:53:01,515 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,515 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,515 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-nodemgr:0?flat
2025-08-11 10:53:01,536 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-nodemgr:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,536 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,536 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-collector:0?flat
2025-08-11 10:53:01,575 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-collector:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,575 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,575 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0?flat
2025-08-11 10:53:01,599 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,599 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,599 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-nodemgr:0?flat
2025-08-11 10:53:01,619 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-nodemgr:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,619 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,619 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-collector:0?flat
2025-08-11 10:53:01,655 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-collector:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,655 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,655 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0?flat
2025-08-11 10:53:01,676 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,677 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,677 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Database:contrail-query-engine:0?flat
2025-08-11 10:53:01,705 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Database:contrail-query-engine:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,705 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,705 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Database:contrail-database-nodemgr:0?flat
2025-08-11 10:53:01,723 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Database:contrail-database-nodemgr:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,723 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,723 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Database:contrail-query-engine:0?flat
2025-08-11 10:53:01,752 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Database:contrail-query-engine:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,753 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,753 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Database:contrail-database-nodemgr:0?flat
2025-08-11 10:53:01,775 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Database:contrail-database-nodemgr:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,776 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,776 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Database:contrail-query-engine:0?flat
2025-08-11 10:53:01,810 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Database:contrail-query-engine:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,810 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,810 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Database:contrail-database-nodemgr:0?flat
2025-08-11 10:53:01,835 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Database:contrail-database-nodemgr:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,835 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,835 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-api:0?flat
2025-08-11 10:53:01,857 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-api:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,857 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,857 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Config:contrail-api:0?flat
2025-08-11 10:53:01,880 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Config:contrail-api:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,880 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,880 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Config:contrail-api:0?flat
2025-08-11 10:53:01,901 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Config:contrail-api:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,901 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,901 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-control:0?flat
2025-08-11 10:53:01,949 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-control:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,949 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,949 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-control-nodemgr:0?flat
2025-08-11 10:53:01,970 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-control-nodemgr:0 is connected to collector 10.0.0.56
2025-08-11 10:53:01,970 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:01,970 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-dns:0?flat
2025-08-11 10:53:02,000 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-dns:0 is connected to collector 10.0.0.56
2025-08-11 10:53:02,000 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:02,001 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-control:0?flat
2025-08-11 10:53:02,052 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-control:0 is connected to collector 10.0.0.56
2025-08-11 10:53:02,052 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:02,052 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-control-nodemgr:0?flat
2025-08-11 10:53:02,071 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-control-nodemgr:0 is connected to collector 10.0.0.56
2025-08-11 10:53:02,071 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:02,071 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-dns:0?flat
2025-08-11 10:53:02,105 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-dns:0 is connected to collector 10.0.0.56
2025-08-11 10:53:02,105 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:02,105 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-control:0?flat
2025-08-11 10:53:02,163 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-control:0 is connected to collector 10.0.0.56
2025-08-11 10:53:02,163 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:02,163 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-control-nodemgr:0?flat
2025-08-11 10:53:02,183 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-control-nodemgr:0 is connected to collector 10.0.0.56
2025-08-11 10:53:02,183 - DEBUG - Verifying through opserver in 10.0.0.56
2025-08-11 10:53:02,183 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-dns:0?flat
2025-08-11 10:53:02,216 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-dns:0 is connected to collector 10.0.0.56
2025-08-11 10:53:02,809 - DEBUG - No XMPP flaps were noticed during the test
2025-08-11 10:53:02,810 - INFO - END TEST : test_verify_generator_connections_to_collector_node : PASSED[0:00:02]
2025-08-11 10:53:02,810 - INFO - --------------------------------------------------------------------------------
2025-08-11 10:53:02,813 - INFO - ================================================================================
2025-08-11 10:53:02,813 - INFO - STARTING TEST : test_verify_process_status_agent
2025-08-11 10:53:02,813 - INFO - TEST DESCRIPTION : Test to validate process_status
2025-08-11 10:53:03,417 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with
2025-08-11 10:53:03,417 - INFO - Initial checks done. Running the testcase now
2025-08-11 10:53:03,417 - INFO -
2025-08-11 10:53:03,418 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3803-1?flat
2025-08-11 10:53:03,454 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.20.0.17:5269'] servers UP
2025-08-11 10:53:03,454 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1 connected to xmpp 10.20.0.17
2025-08-11 10:53:03,455 - WARNING - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.20.0.254:5269'] servers NOT UP
2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.20.0.25:5269'] servers UP
2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1 connected to xmpp 10.20.0.25
2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.20.0.17:53'] servers UP
2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1 connected to dns 10.20.0.17
2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.20.0.254:53'] servers UP
2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1 connected to dns 10.20.0.254
2025-08-11 10:53:03,455 - WARNING - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.20.0.25:53'] servers NOT UP
2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.0.0.56:8086'] servers UP
2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1 connected to collector 10.0.0.56
2025-08-11 10:53:03,455 - WARNING - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.0.0.38:8086'] servers NOT UP
2025-08-11 10:53:03,456 - WARNING - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.0.0.22:8086'] servers NOT UP
2025-08-11 10:53:04,070 - DEBUG - No XMPP flaps were noticed during the test
2025-08-11 10:53:04,070 - INFO - END TEST : test_verify_process_status_agent : PASSED[0:00:02]
2025-08-11 10:53:04,070 - INFO - --------------------------------------------------------------------------------
2025-08-11 10:53:04,073 - INFO - ================================================================================
2025-08-11 10:53:04,073 - INFO - STARTING TEST : test_verify_process_status_analytics_node
2025-08-11 10:53:04,073 - INFO - TEST DESCRIPTION : Test to validate process_status-Analytics-Node
2025-08-11 10:53:04,691 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with
2025-08-11 10:53:04,691 - INFO - Initial checks done. Running the testcase now
2025-08-11 10:53:04,691 - INFO -
2025-08-11 10:53:04,691 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 10:53:04,776 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat
2025-08-11 10:53:04,814 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat
2025-08-11 10:53:04,867 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 10:53:04,911 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-collector module connection to ['10.0.0.56:8086'] servers UP
2025-08-11 10:53:04,912 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-collector module connection to ['10.0.0.56:6379'] servers NOT UP
2025-08-11 10:53:04,912 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-collector module connection to ['10.0.0.38:6379'] servers NOT UP
2025-08-11 10:53:04,912 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-collector module connection to ['10.0.0.22:6379'] servers NOT UP
2025-08-11 10:53:04,912 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-collector module connection to ['127.0.0.1:6379'] servers UP
2025-08-11 10:53:04,912 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-collector module connection to ['10.0.0.56:9042'] servers UP
2025-08-11 10:53:04,912 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat
2025-08-11 10:53:04,950 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-collector module connection to ['10.0.0.56:8086'] servers NOT UP
2025-08-11 10:53:04,950 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-collector module connection to ['10.0.0.38:8086'] servers UP
2025-08-11 10:53:04,950 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-collector module connection to ['10.0.0.56:6379'] servers NOT UP
2025-08-11 10:53:04,950 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-collector module connection to ['10.0.0.38:6379'] servers NOT UP
2025-08-11 10:53:04,950 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-collector module connection to ['10.0.0.22:6379'] servers NOT UP
2025-08-11 10:53:04,950 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-collector module connection to ['127.0.0.1:6379'] servers UP
2025-08-11 10:53:04,950 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-collector module connection to ['10.0.0.56:9042'] servers UP
2025-08-11 10:53:04,950 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat
2025-08-11 10:53:04,985 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['10.0.0.56:8086'] servers NOT UP
2025-08-11 10:53:04,985 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['10.0.0.38:8086'] servers NOT UP
2025-08-11 10:53:04,985 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['10.0.0.22:8086'] servers UP
2025-08-11 10:53:04,985 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['10.0.0.56:6379'] servers NOT UP
2025-08-11 10:53:04,986 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['10.0.0.38:6379'] servers NOT UP
2025-08-11 10:53:04,986 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['10.0.0.22:6379'] servers NOT UP
2025-08-11 10:53:04,986 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['127.0.0.1:6379'] servers UP
2025-08-11 10:53:04,986 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['10.0.0.56:9042'] servers UP
2025-08-11 10:53:04,986 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 10:53:05,026 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-analytics-api module connection to ['10.0.0.56:8086'] servers UP
2025-08-11 10:53:05,027 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat
2025-08-11 10:53:05,059 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-analytics-api module connection to ['10.0.0.56:8086'] servers UP
2025-08-11 10:53:05,059 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat
2025-08-11 10:53:05,098 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-analytics-api module connection to ['10.0.0.56:8086'] servers NOT UP
2025-08-11 10:53:05,098 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-analytics-api module connection to ['10.0.0.38:8086'] servers NOT UP
2025-08-11 10:53:05,098 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-analytics-api module connection to ['10.0.0.22:8086'] servers UP
2025-08-11 10:53:05,098 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 10:53:05,139 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-analytics-nodemgr module connection to ['10.0.0.56:8086'] servers NOT UP
2025-08-11 10:53:05,139 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-analytics-nodemgr module connection to ['10.0.0.38:8086'] servers UP
2025-08-11 10:53:05,139 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat
2025-08-11 10:53:05,172 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-analytics-nodemgr module connection to ['10.0.0.56:8086'] servers NOT UP
2025-08-11 10:53:05,172 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-analytics-nodemgr module connection to ['10.0.0.38:8086'] servers UP
2025-08-11 10:53:05,172 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat
2025-08-11 10:53:05,208 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-analytics-nodemgr module connection to ['10.0.0.56:8086'] servers NOT UP
2025-08-11 10:53:05,208 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-analytics-nodemgr module connection to ['10.0.0.38:8086'] servers NOT UP
2025-08-11 10:53:05,208 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-analytics-nodemgr module connection to ['10.0.0.22:8086'] servers UP
2025-08-11 10:53:05,812 - DEBUG - No XMPP flaps were noticed during the test
2025-08-11 10:53:05,812 - INFO - END TEST : test_verify_process_status_analytics_node : PASSED[0:00:01]
2025-08-11 10:53:05,812 - INFO - --------------------------------------------------------------------------------
2025-08-11 10:53:05,815 - INFO - ================================================================================
2025-08-11 10:53:05,815 - INFO - STARTING TEST : test_verify_process_status_config
2025-08-11 10:53:05,815 - INFO - TEST DESCRIPTION : Test to validate process_status-Config
2025-08-11 10:53:06,430 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with
2025-08-11 10:53:06,430 - INFO - Initial checks done. Running the testcase now
2025-08-11 10:53:06,430 - INFO -
2025-08-11 10:53:06,430 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 10:53:06,465 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat
2025-08-11 10:53:06,490 - WARNING - No object found for module contrail-device-manager
2025-08-11 10:53:07,113 - ERROR - AssertionError
Python 3.6.8: /usr/bin/python3
Mon Aug 11 10:53:06 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={})
78 log.info('Initial checks done. Running the testcase now')
79 log.info('')
80 result = function(self, *args, **kwargs)
81 if self.inputs.upgrade:
82 pid = os.getpid()
result = None
function =
self =
args = ()
kwargs = {}
/contrail-test/scripts/analytics/test_analytics.py in test_verify_process_status_config(self=)
333
334 '''
335 self.analytics_obj.verify_process_and_connection_infos_config()
336
337 @test.attr(type=['cb_sanity', 'sanity', 'dev_sanity_dpdk'])
self =
self.analytics_obj =
self.analytics_obj.verify_process_and_connection_infos_config = >
/contrail-test/tcutils/collector/analytics_tests.py in verify_process_and_connection_infos_config(self=)
4126 for k in module_connection_dict:
4127 result1 = self.verify_process_status(ops_inspect, k)
4128 assert result1, f'status is invalid for {k}\n' + str(ops_inspect)
4129
4130 cfgm = self.inputs.cfgm_names[0]
result1 = False
builtinstr =
ops_inspect = {'NodeStatus': {'process_status': [{'module_id':...uid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}}
AssertionError: status is invalid for contrail-device-manager
{'NodeStatus': {'process_status': [{'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754909561383898, 'system_mem_usage': {'total': 16175640, 'used': 5809908, 'free': 1998564, 'buffers': 286068, 'cached': 8081100, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 8358584, 'partition_space_available_1k': 72731900, 'percentage_partition_space_used': 10}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 98824, 'cpu_share': 0.0, 'mem_res': 94496}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 103144, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 51992, 'cpu_share': 0.01, 'mem_res': 40096}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.33, 'five_min_avg': 1.48, 'fifteen_min_avg': 0.87, 'cpu_share': 6.75, 'node_type': 'config-node'}, 'deleted': False}, 'ContrailConfig': {'__T': 1754909358195620, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.38"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-2."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-2."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1556493265386885059, "uuid_lslong": 13820473496784322762}, "enable": true, "created": "2025-08-11T10:47:02.163858", "last_modified": "2025-08-11T10:47:02.163858", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}}
__cause__ = None
__class__ =
__context__ = None
__delattr__ =
__dict__ = {}
__dir__ =
__doc__ = 'Assertion failed.'
__eq__ =
__format__ =
__ge__ =
__getattribute__ =
__gt__ =
__hash__ =
__init__ =
__init_subclass__ =
__le__ =
__lt__ =
__ne__ =
__new__ =
__reduce__ =
__reduce_ex__ =
__repr__ =
__setattr__ =
__setstate__ =
__sizeof__ =
__str__ =
__subclasshook__ =
__suppress_context__ = False
__traceback__ =
args = ('status is invalid for contrail-device-manager\n{\'...uid\': \'"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"\'}}}',)
with_traceback =
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
result = function(self, *args, **kwargs)
File "/contrail-test/scripts/analytics/test_analytics.py", line 335, in test_verify_process_status_config
self.analytics_obj.verify_process_and_connection_infos_config()
File "/contrail-test/tcutils/collector/analytics_tests.py", line 4128, in verify_process_and_connection_infos_config
assert result1, f'status is invalid for {k}\n' + str(ops_inspect)
AssertionError: status is invalid for contrail-device-manager
{'NodeStatus': {'process_status': [{'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754909561383898, 'system_mem_usage': {'total': 16175640, 'used': 5809908, 'free': 1998564, 'buffers': 286068, 'cached': 8081100, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 8358584, 'partition_space_available_1k': 72731900, 'percentage_partition_space_used': 10}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 98824, 'cpu_share': 0.0, 'mem_res': 94496}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 103144, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 51992, 'cpu_share': 0.01, 'mem_res': 40096}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.33, 'five_min_avg': 1.48, 'fifteen_min_avg': 0.87, 'cpu_share': 6.75, 'node_type': 'config-node'}, 'deleted': False}, 'ContrailConfig': {'__T': 1754909358195620, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.38"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-2."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-2."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1556493265386885059, "uuid_lslong": 13820473496784322762}, "enable": true, "created": "2025-08-11T10:47:02.163858", "last_modified": "2025-08-11T10:47:02.163858", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}}
2025-08-11 10:53:07,126 - DEBUG - No XMPP flaps were noticed during the test
2025-08-11 10:53:07,126 - INFO -
2025-08-11 10:53:07,126 - INFO - END TEST : test_verify_process_status_config : FAILED[0:00:02]
2025-08-11 10:53:07,126 - INFO - --------------------------------------------------------------------------------
2025-08-11 10:53:07,129 - INFO - ================================================================================
2025-08-11 10:53:07,129 - INFO - STARTING TEST : test_verify_process_status_control_node
2025-08-11 10:53:07,129 - INFO - TEST DESCRIPTION : Test to validate process_status-Control-Node
2025-08-11 10:53:07,725 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with
2025-08-11 10:53:07,726 - INFO - Initial checks done. Running the testcase now
2025-08-11 10:53:07,726 - INFO -
2025-08-11 10:53:07,726 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-1?flat
2025-08-11 10:53:07,747 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1:contrail-control module connection to 10.0.0.56:9041 servers UP
2025-08-11 10:53:07,747 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1:contrail-control module connection to 10.0.0.56:5673 servers UP
2025-08-11 10:53:07,747 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1:contrail-control module connection to ['10.0.0.56:8086'] servers NOT UP
2025-08-11 10:53:07,747 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1:contrail-control module connection to ['10.0.0.38:8086'] servers NOT UP
2025-08-11 10:53:07,747 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1:contrail-control module connection to ['10.0.0.22:8086'] servers UP
2025-08-11 10:53:07,747 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-2?flat
2025-08-11 10:53:07,768 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2:contrail-control module connection to 10.0.0.56:9041 servers UP
2025-08-11 10:53:07,768 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2:contrail-control module connection to 10.0.0.56:5673 servers UP
2025-08-11 10:53:07,768 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-2:contrail-control module connection to ['10.0.0.56:8086'] servers NOT UP
2025-08-11 10:53:07,768 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-2:contrail-control module connection to ['10.0.0.38:8086'] servers NOT UP
2025-08-11 10:53:07,768 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2:contrail-control module connection to ['10.0.0.22:8086'] servers UP
2025-08-11 10:53:07,768 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-3?flat
2025-08-11 10:53:07,788 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3:contrail-control module connection to 10.0.0.56:9041 servers UP
2025-08-11 10:53:07,788 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3:contrail-control module connection to 10.0.0.56:5673 servers UP
2025-08-11 10:53:07,788 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3:contrail-control module connection to ['10.0.0.56:8086'] servers NOT UP
2025-08-11 10:53:07,788 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3:contrail-control module connection to ['10.0.0.38:8086'] servers NOT UP
2025-08-11 10:53:07,788 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3:contrail-control module connection to ['10.0.0.22:8086'] servers UP
2025-08-11 10:53:08,396 - DEBUG - No XMPP flaps were noticed during the test
2025-08-11 10:53:08,396 - INFO - END TEST : test_verify_process_status_control_node : PASSED[0:00:01]
2025-08-11 10:53:08,397 - INFO - --------------------------------------------------------------------------------
2025-08-11 12:34:43,662 - DEBUG - [10.0.0.56]: Running cmd : hostname
2025-08-11 12:34:43,859 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1
2025-08-11 12:34:43,859 - DEBUG - [10.0.0.56]: Running cmd : hostname -f
2025-08-11 12:34:43,918 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 12:34:43,918 - DEBUG - [10.0.0.56]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:34:44,007 - DEBUG - Output : NAMES
contrail_test_izHBaL8iO
analytics_snmp_nodemgr_1
analytics_snmp_topology_1
analytics_snmp_snmp-collector_1
analytics_alarm_alarm-gen_1
analytics_alarm_nodemgr_1
analytics_alarm_kafka_1
analytics_nodemgr_1
analytics_collector_1
analytics_api_1
analytics_database_query-engine_1
analytics_database_nodemgr_1
analytics_database_cassandra_1
control_named_1
control_control_1
control_nodemgr_1
control_dns_1
webui_job_1
webui_web_1
config_devicemgr_1
config_api_1
config_schema_1
config_dnsmasq_1
config_nodemgr_1
config_svcmonitor_1
config_database_rabbitmq_1
config_database_nodemgr_1
config_database_zookeeper_1
config_database_cassandra_1
redis_redis_1
barbican_worker
barbican_keystone_listener
barbican_api
horizon
heat_engine
heat_api_cfn
heat_api
neutron_server
nova_novncproxy
nova_conductor
nova_api
nova_scheduler
glance_api
keystone
keystone_fernet
keystone_ssh
rabbitmq
memcached
mariadb
cron
kolla_toolbox
fluentd
2025-08-11 12:34:44,007 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.0.0.56 | head -n 1 | awk '{print $2}'
2025-08-11 12:34:44,066 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 12:34:44,067 - DEBUG - [10.0.0.56]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-08-11 12:34:44,127 - DEBUG - Output : 127.0.0.1/8
10.0.0.56/24
10.20.0.17/24
172.17.0.1/16
2025-08-11 12:34:44,127 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.20.0.17 | head -n 1 | awk '{print $2}'
2025-08-11 12:34:44,186 - DEBUG - Output :
2025-08-11 12:34:44,186 - DEBUG - [10.0.0.38]: Running cmd : hostname
2025-08-11 12:34:44,363 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 12:34:44,363 - DEBUG - [10.0.0.38]: Running cmd : hostname -f
2025-08-11 12:34:44,421 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 12:34:44,422 - DEBUG - [10.0.0.38]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:34:44,513 - DEBUG - Output : NAMES
analytics_snmp_snmp-collector_1
analytics_snmp_nodemgr_1
analytics_snmp_topology_1
analytics_alarm_alarm-gen_1
analytics_alarm_nodemgr_1
analytics_alarm_kafka_1
analytics_nodemgr_1
analytics_api_1
analytics_collector_1
analytics_database_nodemgr_1
analytics_database_query-engine_1
analytics_database_cassandra_1
control_named_1
control_dns_1
control_nodemgr_1
control_control_1
webui_web_1
webui_job_1
config_devicemgr_1
config_api_1
config_dnsmasq_1
config_schema_1
config_svcmonitor_1
config_nodemgr_1
config_database_nodemgr_1
config_database_rabbitmq_1
config_database_cassandra_1
config_database_zookeeper_1
redis_redis_1
2025-08-11 12:34:44,513 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-08-11 12:34:44,578 - DEBUG - Output : 127.0.0.1/8
10.0.0.38/24
10.20.0.254/24
172.17.0.1/16
2025-08-11 12:34:44,578 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.0.0.38 | head -n 1 | awk '{print $2}'
2025-08-11 12:34:44,638 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 12:34:44,638 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.20.0.254 | head -n 1 | awk '{print $2}'
2025-08-11 12:34:49,713 - DEBUG - Output :
2025-08-11 12:34:49,713 - DEBUG - [10.0.0.22]: Running cmd : hostname
2025-08-11 12:34:49,884 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3
2025-08-11 12:34:49,884 - DEBUG - [10.0.0.22]: Running cmd : hostname -f
2025-08-11 12:34:49,940 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3.
2025-08-11 12:34:49,940 - DEBUG - [10.0.0.22]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:34:50,018 - DEBUG - Output : NAMES
analytics_snmp_snmp-collector_1
analytics_snmp_topology_1
analytics_snmp_nodemgr_1
analytics_alarm_nodemgr_1
analytics_alarm_alarm-gen_1
analytics_alarm_kafka_1
analytics_nodemgr_1
analytics_api_1
analytics_collector_1
analytics_database_query-engine_1
analytics_database_nodemgr_1
analytics_database_cassandra_1
control_named_1
control_nodemgr_1
control_dns_1
control_control_1
webui_web_1
webui_job_1
config_devicemgr_1
config_nodemgr_1
config_svcmonitor_1
config_api_1
config_schema_1
config_dnsmasq_1
config_database_rabbitmq_1
config_database_nodemgr_1
config_database_zookeeper_1
config_database_cassandra_1
redis_redis_1
2025-08-11 12:34:50,018 - DEBUG - [10.0.0.22]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-08-11 12:34:50,081 - DEBUG - Output : 127.0.0.1/8
10.0.0.22/24
10.20.0.25/24
172.17.0.1/16
2025-08-11 12:34:50,081 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.0.0.22 | head -n 1 | awk '{print $2}'
2025-08-11 12:34:50,141 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3.
2025-08-11 12:34:50,141 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.20.0.25 | head -n 1 | awk '{print $2}'
2025-08-11 12:34:50,201 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3
2025-08-11 12:34:50,201 - DEBUG - [10.0.0.19]: Running cmd : hostname
2025-08-11 12:34:50,413 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1
2025-08-11 12:34:50,414 - DEBUG - [10.0.0.19]: Running cmd : hostname -f
2025-08-11 12:34:50,472 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 12:34:50,472 - DEBUG - [10.0.0.19]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:34:50,553 - DEBUG - Output : NAMES
vrouter_vrouter-agent_1
vrouter_nodemgr_1
rsyslogd_rsyslogd_1
nova_compute
nova_libvirt
nova_ssh
cron
kolla_toolbox
fluentd
2025-08-11 12:34:50,554 - DEBUG - [10.0.0.19]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-08-11 12:34:50,612 - DEBUG - Output : 10.20.0.193/24
2025-08-11 12:34:50,612 - DEBUG - [10.0.0.19]: Running cmd : getent hosts 10.20.0.193 | head -n 1 | awk '{print $2}'
2025-08-11 12:34:50,687 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1
2025-08-11 12:34:50,687 - DEBUG - [10.0.0.41]: Running cmd : hostname
2025-08-11 12:34:50,858 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 12:34:50,858 - DEBUG - [10.0.0.41]: Running cmd : hostname -f
2025-08-11 12:34:50,919 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 12:34:50,919 - DEBUG - [10.0.0.41]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:34:50,999 - DEBUG - Output : NAMES
vrouter_vrouter-agent_1
vrouter_nodemgr_1
rsyslogd_rsyslogd_1
nova_compute
nova_libvirt
nova_ssh
cron
kolla_toolbox
fluentd
2025-08-11 12:34:50,999 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-08-11 12:34:51,058 - DEBUG - Output : 10.20.0.129/24
2025-08-11 12:34:51,058 - DEBUG - [10.0.0.41]: Running cmd : getent hosts 10.20.0.129 | head -n 1 | awk '{print $2}'
2025-08-11 12:34:51,123 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 12:34:51,959 - DEBUG - Not creating keypair since it exists
2025-08-11 12:34:52,323 - INFO - Using existing project ['default-domain', 'admin'](74e57a64-053a-4ab9-9a41-98124cb3c978)
2025-08-11 12:34:52,494 - INFO - ================================================================================
2025-08-11 12:34:52,494 - INFO - STARTING TEST : test_verify_process_status_config
2025-08-11 12:34:52,495 - INFO - TEST DESCRIPTION : Test to validate process_status-Config
2025-08-11 12:34:53,093 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:34:53,100 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:34:53,106 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with
2025-08-11 12:34:53,106 - INFO - Initial checks done. Running the testcase now
2025-08-11 12:34:53,106 - INFO -
2025-08-11 12:34:53,106 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 12:34:53,396 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 12:34:53,458 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat
2025-08-11 12:34:53,517 - WARNING - No object found for module contrail-device-manager
2025-08-11 12:34:54,120 - ERROR - AssertionError
Python 3.6.8: /usr/bin/python3
Mon Aug 11 12:34:53 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={})
78 log.info('Initial checks done. Running the testcase now')
79 log.info('')
80 result = function(self, *args, **kwargs)
81 if self.inputs.upgrade:
82 pid = os.getpid()
result = None
function =
self =
args = ()
kwargs = {}
/contrail-test/scripts/analytics/test_analytics.py in test_verify_process_status_config(self=)
333
334 '''
335 self.analytics_obj.verify_process_and_connection_infos_config()
336
337 @test.attr(type=['cb_sanity', 'sanity', 'dev_sanity_dpdk'])
self =
self.analytics_obj =
self.analytics_obj.verify_process_and_connection_infos_config = >
/contrail-test/tcutils/collector/analytics_tests.py in verify_process_and_connection_infos_config(self=)
4126 for k in module_connection_dict:
4127 result1 = self.verify_process_status(ops_inspect, k)
4128 assert result1, f'status is invalid for {k}\n' + str(ops_inspect)
4129
4130 cfgm = self.inputs.cfgm_names[0]
result1 = False
builtinstr =
ops_inspect = {'NodeStatus': {'process_status': [{'module_id':...uid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}}
AssertionError: status is invalid for contrail-device-manager
{'NodeStatus': {'process_status': [{'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754915682118270, 'system_mem_usage': {'total': 16175640, 'used': 6773708, 'free': 233396, 'buffers': 291184, 'cached': 8877352, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9153532, 'partition_space_available_1k': 71936952, 'percentage_partition_space_used': 11}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 134472, 'cpu_share': 0.01, 'mem_res': 109176}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 103136, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 52260, 'cpu_share': 0.01, 'mem_res': 40192}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.06, 'five_min_avg': 1.31, 'fifteen_min_avg': 1.43, 'cpu_share': 7.0, 'node_type': 'config-node'}, 'deleted': False}, 'ContrailConfig': {'__T': 1754909358195620, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.38"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-2."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-2."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1556493265386885059, "uuid_lslong": 13820473496784322762}, "enable": true, "created": "2025-08-11T10:47:02.163858", "last_modified": "2025-08-11T10:47:02.163858", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}}
__cause__ = None
__class__ =
__context__ = None
__delattr__ =
__dict__ = {}
__dir__ =
__doc__ = 'Assertion failed.'
__eq__ =
__format__ =
__ge__ =
__getattribute__ =
__gt__ =
__hash__ =
__init__ =
__init_subclass__ =
__le__ =
__lt__ =
__ne__ =
__new__ =
__reduce__ =
__reduce_ex__ =
__repr__ =
__setattr__ =
__setstate__ =
__sizeof__ =
__str__ =
__subclasshook__ =
__suppress_context__ = False
__traceback__ =
args = ('status is invalid for contrail-device-manager\n{\'...uid\': \'"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"\'}}}',)
with_traceback =
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
result = function(self, *args, **kwargs)
File "/contrail-test/scripts/analytics/test_analytics.py", line 335, in test_verify_process_status_config
self.analytics_obj.verify_process_and_connection_infos_config()
File "/contrail-test/tcutils/collector/analytics_tests.py", line 4128, in verify_process_and_connection_infos_config
assert result1, f'status is invalid for {k}\n' + str(ops_inspect)
AssertionError: status is invalid for contrail-device-manager
{'NodeStatus': {'process_status': [{'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754915682118270, 'system_mem_usage': {'total': 16175640, 'used': 6773708, 'free': 233396, 'buffers': 291184, 'cached': 8877352, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9153532, 'partition_space_available_1k': 71936952, 'percentage_partition_space_used': 11}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 134472, 'cpu_share': 0.01, 'mem_res': 109176}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 103136, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 52260, 'cpu_share': 0.01, 'mem_res': 40192}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.06, 'five_min_avg': 1.31, 'fifteen_min_avg': 1.43, 'cpu_share': 7.0, 'node_type': 'config-node'}, 'deleted': False}, 'ContrailConfig': {'__T': 1754909358195620, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.38"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-2."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-2."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1556493265386885059, "uuid_lslong": 13820473496784322762}, "enable": true, "created": "2025-08-11T10:47:02.163858", "last_modified": "2025-08-11T10:47:02.163858", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}}
2025-08-11 12:34:54,121 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:34:54,128 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:34:54,134 - DEBUG - No XMPP flaps were noticed during the test
2025-08-11 12:34:54,134 - INFO -
2025-08-11 12:34:54,134 - INFO - END TEST : test_verify_process_status_config : FAILED[0:00:02]
2025-08-11 12:34:54,134 - INFO - --------------------------------------------------------------------------------
2025-08-11 12:39:02,931 - DEBUG - [10.0.0.56]: Running cmd : hostname
2025-08-11 12:39:03,123 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1
2025-08-11 12:39:03,123 - DEBUG - [10.0.0.56]: Running cmd : hostname -f
2025-08-11 12:39:03,182 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 12:39:03,182 - DEBUG - [10.0.0.56]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:39:03,327 - DEBUG - Output : NAMES
contrail_test_izHBaL8iO
analytics_snmp_nodemgr_1
analytics_snmp_topology_1
analytics_snmp_snmp-collector_1
analytics_alarm_alarm-gen_1
analytics_alarm_nodemgr_1
analytics_alarm_kafka_1
analytics_nodemgr_1
analytics_collector_1
analytics_api_1
analytics_database_query-engine_1
analytics_database_nodemgr_1
analytics_database_cassandra_1
control_named_1
control_control_1
control_nodemgr_1
control_dns_1
webui_job_1
webui_web_1
config_devicemgr_1
config_api_1
config_schema_1
config_dnsmasq_1
config_nodemgr_1
config_svcmonitor_1
config_database_rabbitmq_1
config_database_nodemgr_1
config_database_zookeeper_1
config_database_cassandra_1
redis_redis_1
barbican_worker
barbican_keystone_listener
barbican_api
horizon
heat_engine
heat_api_cfn
heat_api
neutron_server
nova_novncproxy
nova_conductor
nova_api
nova_scheduler
glance_api
keystone
keystone_fernet
keystone_ssh
rabbitmq
memcached
mariadb
cron
kolla_toolbox
fluentd
2025-08-11 12:39:03,328 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.0.0.56 | head -n 1 | awk '{print $2}'
2025-08-11 12:39:03,382 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 12:39:03,383 - DEBUG - [10.0.0.56]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-08-11 12:39:03,442 - DEBUG - Output : 127.0.0.1/8
10.0.0.56/24
10.20.0.17/24
172.17.0.1/16
2025-08-11 12:39:03,442 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.20.0.17 | head -n 1 | awk '{print $2}'
2025-08-11 12:39:03,502 - DEBUG - Output :
2025-08-11 12:39:03,502 - DEBUG - [10.0.0.38]: Running cmd : hostname
2025-08-11 12:39:03,683 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 12:39:03,683 - DEBUG - [10.0.0.38]: Running cmd : hostname -f
2025-08-11 12:39:03,741 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 12:39:03,742 - DEBUG - [10.0.0.38]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:39:03,822 - DEBUG - Output : NAMES
analytics_snmp_snmp-collector_1
analytics_snmp_nodemgr_1
analytics_snmp_topology_1
analytics_alarm_alarm-gen_1
analytics_alarm_nodemgr_1
analytics_alarm_kafka_1
analytics_nodemgr_1
analytics_api_1
analytics_collector_1
analytics_database_nodemgr_1
analytics_database_query-engine_1
analytics_database_cassandra_1
control_named_1
control_dns_1
control_nodemgr_1
control_control_1
webui_web_1
webui_job_1
config_devicemgr_1
config_api_1
config_dnsmasq_1
config_schema_1
config_svcmonitor_1
config_nodemgr_1
config_database_nodemgr_1
config_database_rabbitmq_1
config_database_cassandra_1
config_database_zookeeper_1
redis_redis_1
2025-08-11 12:39:03,822 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-08-11 12:39:03,886 - DEBUG - Output : 127.0.0.1/8
10.0.0.38/24
10.20.0.254/24
172.17.0.1/16
2025-08-11 12:39:03,886 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.0.0.38 | head -n 1 | awk '{print $2}'
2025-08-11 12:39:03,945 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 12:39:03,946 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.20.0.254 | head -n 1 | awk '{print $2}'
2025-08-11 12:39:04,097 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 12:39:04,098 - DEBUG - [10.0.0.22]: Running cmd : hostname
2025-08-11 12:39:04,278 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3
2025-08-11 12:39:04,278 - DEBUG - [10.0.0.22]: Running cmd : hostname -f
2025-08-11 12:39:04,337 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3.
2025-08-11 12:39:04,337 - DEBUG - [10.0.0.22]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:39:04,418 - DEBUG - Output : NAMES
analytics_snmp_snmp-collector_1
analytics_snmp_topology_1
analytics_snmp_nodemgr_1
analytics_alarm_nodemgr_1
analytics_alarm_alarm-gen_1
analytics_alarm_kafka_1
analytics_nodemgr_1
analytics_api_1
analytics_collector_1
analytics_database_query-engine_1
analytics_database_nodemgr_1
analytics_database_cassandra_1
control_named_1
control_nodemgr_1
control_dns_1
control_control_1
webui_web_1
webui_job_1
config_devicemgr_1
config_nodemgr_1
config_svcmonitor_1
config_api_1
config_schema_1
config_dnsmasq_1
config_database_rabbitmq_1
config_database_nodemgr_1
config_database_zookeeper_1
config_database_cassandra_1
redis_redis_1
2025-08-11 12:39:04,418 - DEBUG - [10.0.0.22]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-08-11 12:39:04,481 - DEBUG - Output : 127.0.0.1/8
10.0.0.22/24
10.20.0.25/24
172.17.0.1/16
2025-08-11 12:39:04,481 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.0.0.22 | head -n 1 | awk '{print $2}'
2025-08-11 12:39:04,540 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3.
2025-08-11 12:39:04,540 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.20.0.25 | head -n 1 | awk '{print $2}'
2025-08-11 12:39:04,615 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3
2025-08-11 12:39:04,615 - DEBUG - [10.0.0.19]: Running cmd : hostname
2025-08-11 12:39:04,786 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1
2025-08-11 12:39:04,786 - DEBUG - [10.0.0.19]: Running cmd : hostname -f
2025-08-11 12:39:04,848 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 12:39:04,848 - DEBUG - [10.0.0.19]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:39:04,929 - DEBUG - Output : NAMES
vrouter_vrouter-agent_1
vrouter_nodemgr_1
rsyslogd_rsyslogd_1
nova_compute
nova_libvirt
nova_ssh
cron
kolla_toolbox
fluentd
2025-08-11 12:39:04,929 - DEBUG - [10.0.0.19]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-08-11 12:39:04,985 - DEBUG - Output : 10.20.0.193/24
2025-08-11 12:39:04,985 - DEBUG - [10.0.0.19]: Running cmd : getent hosts 10.20.0.193 | head -n 1 | awk '{print $2}'
2025-08-11 12:39:05,044 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1
2025-08-11 12:39:05,044 - DEBUG - [10.0.0.41]: Running cmd : hostname
2025-08-11 12:39:05,211 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 12:39:05,211 - DEBUG - [10.0.0.41]: Running cmd : hostname -f
2025-08-11 12:39:05,271 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 12:39:05,271 - DEBUG - [10.0.0.41]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:39:05,351 - DEBUG - Output : NAMES
vrouter_vrouter-agent_1
vrouter_nodemgr_1
rsyslogd_rsyslogd_1
nova_compute
nova_libvirt
nova_ssh
cron
kolla_toolbox
fluentd
2025-08-11 12:39:05,352 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-08-11 12:39:05,415 - DEBUG - Output : 10.20.0.129/24
2025-08-11 12:39:05,415 - DEBUG - [10.0.0.41]: Running cmd : getent hosts 10.20.0.129 | head -n 1 | awk '{print $2}'
2025-08-11 12:39:05,479 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 12:39:06,470 - DEBUG - Not creating keypair since it exists
2025-08-11 12:39:07,029 - INFO - Using existing project ['default-domain', 'admin'](74e57a64-053a-4ab9-9a41-98124cb3c978)
2025-08-11 12:39:07,317 - INFO - ================================================================================
2025-08-11 12:39:07,317 - INFO - STARTING TEST : test_verify_process_status_config
2025-08-11 12:39:07,319 - INFO - TEST DESCRIPTION : Test to validate process_status-Config
2025-08-11 12:39:07,961 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:39:07,984 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:39:07,990 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.254': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.17': '0'}} with
2025-08-11 12:39:07,990 - INFO - Initial checks done. Running the testcase now
2025-08-11 12:39:07,990 - INFO -
2025-08-11 12:39:07,991 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 12:39:08,520 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 12:39:08,633 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat
2025-08-11 12:39:08,778 - WARNING - No object found for module contrail-device-manager
2025-08-11 12:39:09,409 - ERROR - AssertionError
Python 3.6.8: /usr/bin/python3
Mon Aug 11 12:39:08 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={})
78 log.info('Initial checks done. Running the testcase now')
79 log.info('')
80 result = function(self, *args, **kwargs)
81 if self.inputs.upgrade:
82 pid = os.getpid()
result = None
function =
self =
args = ()
kwargs = {}
/contrail-test/scripts/analytics/test_analytics.py in test_verify_process_status_config(self=)
333
334 '''
335 self.analytics_obj.verify_process_and_connection_infos_config()
336
337 @test.attr(type=['cb_sanity', 'sanity', 'dev_sanity_dpdk'])
self =
self.analytics_obj =
self.analytics_obj.verify_process_and_connection_infos_config = >
/contrail-test/tcutils/collector/analytics_tests.py in verify_process_and_connection_infos_config(self=)
4126 for k in module_connection_dict:
4127 result1 = self.verify_process_status(ops_inspect, k)
4128 assert result1, f'status is invalid for {cfgm} in {k}\n' + str(ops_inspect)
4129
4130 cfgm = self.inputs.cfgm_names[0]
result1 = False
builtinstr =
ops_inspect = {'NodeStatus': {'process_status': [{'module_id':...uid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}}
AssertionError: status is invalid for cn-jenkins-deploy-platform-ansible-os-3803-2. in contrail-device-manager
{'NodeStatus': {'process_status': [{'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754915922114140, 'system_mem_usage': {'total': 16175640, 'used': 6808980, 'free': 223664, 'buffers': 289416, 'cached': 8853580, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9171164, 'partition_space_available_1k': 71919320, 'percentage_partition_space_used': 11}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 135256, 'cpu_share': 0.02, 'mem_res': 109680}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 103112, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 52260, 'cpu_share': 0.01, 'mem_res': 40192}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.01, 'five_min_avg': 1.14, 'fifteen_min_avg': 1.33, 'cpu_share': 7.25, 'node_type': 'config-node'}, 'deleted': False}, 'ContrailConfig': {'__T': 1754909358195620, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.38"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-2."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-2."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1556493265386885059, "uuid_lslong": 13820473496784322762}, "enable": true, "created": "2025-08-11T10:47:02.163858", "last_modified": "2025-08-11T10:47:02.163858", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}}
__cause__ = None
__class__ =
__context__ = None
__delattr__ =
__dict__ = {}
__dir__ =
__doc__ = 'Assertion failed.'
__eq__ =
__format__ =
__ge__ =
__getattribute__ =
__gt__ =
__hash__ =
__init__ =
__init_subclass__ =
__le__ =
__lt__ =
__ne__ =
__new__ =
__reduce__ =
__reduce_ex__ =
__repr__ =
__setattr__ =
__setstate__ =
__sizeof__ =
__str__ =
__subclasshook__ =
__suppress_context__ = False
__traceback__ =
args = ('status is invalid for cn-jenkins-deploy-platform...uid\': \'"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"\'}}}',)
with_traceback =
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
result = function(self, *args, **kwargs)
File "/contrail-test/scripts/analytics/test_analytics.py", line 335, in test_verify_process_status_config
self.analytics_obj.verify_process_and_connection_infos_config()
File "/contrail-test/tcutils/collector/analytics_tests.py", line 4128, in verify_process_and_connection_infos_config
assert result1, f'status is invalid for {cfgm} in {k}\n' + str(ops_inspect)
AssertionError: status is invalid for cn-jenkins-deploy-platform-ansible-os-3803-2. in contrail-device-manager
{'NodeStatus': {'process_status': [{'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754915922114140, 'system_mem_usage': {'total': 16175640, 'used': 6808980, 'free': 223664, 'buffers': 289416, 'cached': 8853580, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9171164, 'partition_space_available_1k': 71919320, 'percentage_partition_space_used': 11}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 135256, 'cpu_share': 0.02, 'mem_res': 109680}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 103112, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 52260, 'cpu_share': 0.01, 'mem_res': 40192}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.01, 'five_min_avg': 1.14, 'fifteen_min_avg': 1.33, 'cpu_share': 7.25, 'node_type': 'config-node'}, 'deleted': False}, 'ContrailConfig': {'__T': 1754909358195620, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.38"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-2."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-2."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1556493265386885059, "uuid_lslong": 13820473496784322762}, "enable": true, "created": "2025-08-11T10:47:02.163858", "last_modified": "2025-08-11T10:47:02.163858", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}}
2025-08-11 12:39:09,409 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:39:09,416 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:39:09,422 - DEBUG - No XMPP flaps were noticed during the test
2025-08-11 12:39:09,422 - INFO -
2025-08-11 12:39:09,422 - INFO - END TEST : test_verify_process_status_config : FAILED[0:00:02]
2025-08-11 12:39:09,422 - INFO - --------------------------------------------------------------------------------
2025-08-11 12:45:11,916 - DEBUG - [10.0.0.56]: Running cmd : hostname
2025-08-11 12:45:12,107 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1
2025-08-11 12:45:12,108 - DEBUG - [10.0.0.56]: Running cmd : hostname -f
2025-08-11 12:45:12,166 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 12:45:12,167 - DEBUG - [10.0.0.56]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:45:12,282 - DEBUG - Output : NAMES
contrail_test_izHBaL8iO
analytics_snmp_nodemgr_1
analytics_snmp_topology_1
analytics_snmp_snmp-collector_1
analytics_alarm_alarm-gen_1
analytics_alarm_nodemgr_1
analytics_alarm_kafka_1
analytics_nodemgr_1
analytics_collector_1
analytics_api_1
analytics_database_query-engine_1
analytics_database_nodemgr_1
analytics_database_cassandra_1
control_named_1
control_control_1
control_nodemgr_1
control_dns_1
webui_job_1
webui_web_1
config_devicemgr_1
config_api_1
config_schema_1
config_dnsmasq_1
config_nodemgr_1
config_svcmonitor_1
config_database_rabbitmq_1
config_database_nodemgr_1
config_database_zookeeper_1
config_database_cassandra_1
redis_redis_1
barbican_worker
barbican_keystone_listener
barbican_api
horizon
heat_engine
heat_api_cfn
heat_api
neutron_server
nova_novncproxy
nova_conductor
nova_api
nova_scheduler
glance_api
keystone
keystone_fernet
keystone_ssh
rabbitmq
memcached
mariadb
cron
kolla_toolbox
fluentd
2025-08-11 12:45:12,282 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.0.0.56 | head -n 1 | awk '{print $2}'
2025-08-11 12:45:12,352 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 12:45:12,353 - DEBUG - [10.0.0.56]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-08-11 12:45:12,410 - DEBUG - Output : 127.0.0.1/8
10.0.0.56/24
10.20.0.17/24
172.17.0.1/16
2025-08-11 12:45:12,410 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.20.0.17 | head -n 1 | awk '{print $2}'
2025-08-11 12:45:12,471 - DEBUG - Output :
2025-08-11 12:45:12,471 - DEBUG - [10.0.0.38]: Running cmd : hostname
2025-08-11 12:45:12,654 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 12:45:12,655 - DEBUG - [10.0.0.38]: Running cmd : hostname -f
2025-08-11 12:45:12,713 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 12:45:12,714 - DEBUG - [10.0.0.38]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:45:12,805 - DEBUG - Output : NAMES
analytics_snmp_snmp-collector_1
analytics_snmp_nodemgr_1
analytics_snmp_topology_1
analytics_alarm_alarm-gen_1
analytics_alarm_nodemgr_1
analytics_alarm_kafka_1
analytics_nodemgr_1
analytics_api_1
analytics_collector_1
analytics_database_nodemgr_1
analytics_database_query-engine_1
analytics_database_cassandra_1
control_named_1
control_dns_1
control_nodemgr_1
control_control_1
webui_web_1
webui_job_1
config_devicemgr_1
config_api_1
config_dnsmasq_1
config_schema_1
config_svcmonitor_1
config_nodemgr_1
config_database_nodemgr_1
config_database_rabbitmq_1
config_database_cassandra_1
config_database_zookeeper_1
redis_redis_1
2025-08-11 12:45:12,805 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-08-11 12:45:12,868 - DEBUG - Output : 127.0.0.1/8
10.0.0.38/24
10.20.0.254/24
172.17.0.1/16
2025-08-11 12:45:12,868 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.0.0.38 | head -n 1 | awk '{print $2}'
2025-08-11 12:45:12,926 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 12:45:12,927 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.20.0.254 | head -n 1 | awk '{print $2}'
2025-08-11 12:45:17,999 - DEBUG - Output :
2025-08-11 12:45:17,999 - DEBUG - [10.0.0.22]: Running cmd : hostname
2025-08-11 12:45:18,167 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3
2025-08-11 12:45:18,167 - DEBUG - [10.0.0.22]: Running cmd : hostname -f
2025-08-11 12:45:18,224 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3.
2025-08-11 12:45:18,224 - DEBUG - [10.0.0.22]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:45:18,305 - DEBUG - Output : NAMES
analytics_snmp_snmp-collector_1
analytics_snmp_topology_1
analytics_snmp_nodemgr_1
analytics_alarm_nodemgr_1
analytics_alarm_alarm-gen_1
analytics_alarm_kafka_1
analytics_nodemgr_1
analytics_api_1
analytics_collector_1
analytics_database_query-engine_1
analytics_database_nodemgr_1
analytics_database_cassandra_1
control_named_1
control_nodemgr_1
control_dns_1
control_control_1
webui_web_1
webui_job_1
config_devicemgr_1
config_nodemgr_1
config_svcmonitor_1
config_api_1
config_schema_1
config_dnsmasq_1
config_database_rabbitmq_1
config_database_nodemgr_1
config_database_zookeeper_1
config_database_cassandra_1
redis_redis_1
2025-08-11 12:45:18,305 - DEBUG - [10.0.0.22]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-08-11 12:45:18,368 - DEBUG - Output : 127.0.0.1/8
10.0.0.22/24
10.20.0.25/24
172.17.0.1/16
2025-08-11 12:45:18,369 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.0.0.22 | head -n 1 | awk '{print $2}'
2025-08-11 12:45:18,432 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3.
2025-08-11 12:45:18,432 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.20.0.25 | head -n 1 | awk '{print $2}'
2025-08-11 12:45:18,497 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3
2025-08-11 12:45:18,497 - DEBUG - [10.0.0.19]: Running cmd : hostname
2025-08-11 12:45:18,915 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1
2025-08-11 12:45:18,916 - DEBUG - [10.0.0.19]: Running cmd : hostname -f
2025-08-11 12:45:18,972 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 12:45:18,972 - DEBUG - [10.0.0.19]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:45:19,053 - DEBUG - Output : NAMES
vrouter_vrouter-agent_1
vrouter_nodemgr_1
rsyslogd_rsyslogd_1
nova_compute
nova_libvirt
nova_ssh
cron
kolla_toolbox
fluentd
2025-08-11 12:45:19,053 - DEBUG - [10.0.0.19]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-08-11 12:45:19,109 - DEBUG - Output : 10.20.0.193/24
2025-08-11 12:45:19,109 - DEBUG - [10.0.0.19]: Running cmd : getent hosts 10.20.0.193 | head -n 1 | awk '{print $2}'
2025-08-11 12:45:19,168 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1
2025-08-11 12:45:19,168 - DEBUG - [10.0.0.41]: Running cmd : hostname
2025-08-11 12:45:19,338 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 12:45:19,338 - DEBUG - [10.0.0.41]: Running cmd : hostname -f
2025-08-11 12:45:19,400 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 12:45:19,400 - DEBUG - [10.0.0.41]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:45:19,479 - DEBUG - Output : NAMES
vrouter_vrouter-agent_1
vrouter_nodemgr_1
rsyslogd_rsyslogd_1
nova_compute
nova_libvirt
nova_ssh
cron
kolla_toolbox
fluentd
2025-08-11 12:45:19,479 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-08-11 12:45:19,535 - DEBUG - Output : 10.20.0.129/24
2025-08-11 12:45:19,535 - DEBUG - [10.0.0.41]: Running cmd : getent hosts 10.20.0.129 | head -n 1 | awk '{print $2}'
2025-08-11 12:45:19,595 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 12:45:20,406 - DEBUG - Not creating keypair since it exists
2025-08-11 12:45:20,787 - INFO - Using existing project ['default-domain', 'admin'](74e57a64-053a-4ab9-9a41-98124cb3c978)
2025-08-11 12:45:20,965 - INFO - ================================================================================
2025-08-11 12:45:20,966 - INFO - STARTING TEST : test_verify_process_status_config
2025-08-11 12:45:20,966 - INFO - TEST DESCRIPTION : Test to validate process_status-Config
2025-08-11 12:45:21,557 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:45:21,564 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:45:21,570 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.254': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.17': '1'}} with
2025-08-11 12:45:21,570 - INFO - Initial checks done. Running the testcase now
2025-08-11 12:45:21,570 - INFO -
2025-08-11 12:45:21,570 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 12:45:21,859 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 12:45:21,922 - ERROR - cn-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 12:45:21,922 - ERROR - {'NodeStatus': {'process_status': [{'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up'}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-device-manager', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer Connection State updated'}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754916280064757, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 24502700, 'partition_space_available_1k': 56587784, 'percentage_partition_space_used': 30}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 139856, 'cpu_share': 0.0, 'mem_res': 113596}, 'contrail-schema': {'mem_virt': 89560, 'cpu_share': 0.0, 'mem_res': 80084}, 'contrail-svc-monitor': {'mem_virt': 88628, 'cpu_share': 0.0, 'mem_res': 79076}, 'contrail-device-manager': {'mem_virt': 191180, 'cpu_share': 0.0, 'mem_res': 177128}, 'contrail-config-nodemgr': {'mem_virt': 45788, 'cpu_share': 0.01, 'mem_res': 39168}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909187000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'system_cpu_usage': {'one_min_avg': 1.42, 'five_min_avg': 2.02, 'fifteen_min_avg': 2.11, 'cpu_share': 6.25, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175632, 'used': 10317464, 'free': 363332, 'buffers': 94208, 'cached': 5400628, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'ContrailConfig': {'__T': 1754909358195550, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.56"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-1."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-1."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 16855184944691956189, "uuid_lslong": 12031691376148785163}, "enable": true, "created": "2025-08-11T10:47:02.142790", "last_modified": "2025-08-11T10:47:02.142790", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"e9e9a567-93e0-49dd-a6f9-278f5a7aa80b"'}}}
2025-08-11 12:45:21,922 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat
2025-08-11 12:45:22,004 - ERROR - cn-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 12:45:22,004 - ERROR - {'NodeStatus': {'process_status': [{'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754916282181580, 'system_mem_usage': {'total': 16175640, 'used': 6940552, 'free': 251496, 'buffers': 287940, 'cached': 8695652, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9225172, 'partition_space_available_1k': 71865312, 'percentage_partition_space_used': 11}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 139076, 'cpu_share': 0.0, 'mem_res': 111356}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 102876, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 52168, 'cpu_share': 0.01, 'mem_res': 40192}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.29, 'five_min_avg': 1.4, 'fifteen_min_avg': 1.42, 'cpu_share': 6.5, 'node_type': 'config-node'}, 'deleted': False}, 'ContrailConfig': {'__T': 1754909358195620, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.38"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-2."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-2."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1556493265386885059, "uuid_lslong": 13820473496784322762}, "enable": true, "created": "2025-08-11T10:47:02.163858", "last_modified": "2025-08-11T10:47:02.163858", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}}
2025-08-11 12:45:22,004 - WARNING - No object found for module contrail-device-manager
2025-08-11 12:45:22,004 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat
2025-08-11 12:45:22,031 - ERROR - cn-jenkins-deploy-platform-ansible-os-3803-3.
2025-08-11 12:45:22,031 - ERROR - {'NodeStatus': {'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9245104, 'partition_space_available_1k': 71845380, 'percentage_partition_space_used': 11}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 165268, 'cpu_share': 0.0, 'mem_res': 118396}, 'contrail-schema': {'mem_virt': 77756, 'cpu_share': 0.0, 'mem_res': 74824}, 'contrail-svc-monitor': {'mem_virt': 75776, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 102832, 'cpu_share': 0.0, 'mem_res': 84820}, 'contrail-config-nodemgr': {'mem_virt': 52232, 'cpu_share': 0.01, 'mem_res': 40188}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, '__T': 1754916280266459, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'process_status': [{'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], 'system_cpu_usage': {'one_min_avg': 0.92, 'five_min_avg': 1.08, 'fifteen_min_avg': 1.09, 'cpu_share': 5.25, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175640, 'used': 6999768, 'free': 245776, 'buffers': 285784, 'cached': 8644312, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'ContrailConfig': {'__T': 1754909358195771, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.22"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-3."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-3."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12779472725754465482, "uuid_lslong": 11944146654837029108}, "enable": true, "created": "2025-08-11T10:47:02.187565", "last_modified": "2025-08-11T10:47:02.187565", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"b159cebc-4b63-4cca-a5c2-22198abb38f4"'}}}
2025-08-11 12:45:22,031 - WARNING - No object found for module contrail-device-manager
2025-08-11 12:45:22,031 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 12:45:22,059 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.56:8086 servers NOT UP
2025-08-11 12:45:22,059 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.38:8086 servers NOT UP
2025-08-11 12:45:22,059 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.22:8086 servers UP
2025-08-11 12:45:22,059 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.56:2181 servers UP
2025-08-11 12:45:22,060 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.56:9161 servers NOT UP
2025-08-11 12:45:22,060 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.38:9161 servers NOT UP
2025-08-11 12:45:22,060 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.22:9161 servers NOT UP
2025-08-11 12:45:22,060 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to cn-jenkins-deploy-platform-ansible-os-3803-1.*:9161 servers NOT UP
2025-08-11 12:45:22,060 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to cn-jenkins-deploy-platform-ansible-os-3803-2.*:9161 servers NOT UP
2025-08-11 12:45:22,061 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to cn-jenkins-deploy-platform-ansible-os-3803-3.*:9161 servers NOT UP
2025-08-11 12:45:22,653 - ERROR - AssertionError
Python 3.6.8: /usr/bin/python3
Mon Aug 11 12:45:22 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={})
78 log.info('Initial checks done. Running the testcase now')
79 log.info('')
80 result = function(self, *args, **kwargs)
81 if self.inputs.upgrade:
82 pid = os.getpid()
result = None
function =
self =
args = ()
kwargs = {}
/contrail-test/scripts/analytics/test_analytics.py in test_verify_process_status_config(self=)
333
334 '''
335 self.analytics_obj.verify_process_and_connection_infos_config()
336
337 @test.attr(type=['cb_sanity', 'sanity', 'dev_sanity_dpdk'])
self =
self.analytics_obj =
self.analytics_obj.verify_process_and_connection_infos_config = >
/contrail-test/tcutils/collector/analytics_tests.py in verify_process_and_connection_infos_config(self=)
4176 result = result or self.verify_connection_infos(
4177 ops_inspect, 'contrail-api', server, node=cfgm)
4178 assert result, 'contrail-api is not connected to config db\n' + str(ops_inspect)
4179
4180 result = False
result = False
builtinstr =
ops_inspect = {'NodeStatus': {'process_status': [{'module_id':...uid': '"e9e9a567-93e0-49dd-a6f9-278f5a7aa80b"'}}}
AssertionError: contrail-api is not connected to config db
{'NodeStatus': {'process_status': [{'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up'}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-device-manager', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer Connection State updated'}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754916280064757, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 24502700, 'partition_space_available_1k': 56587784, 'percentage_partition_space_used': 30}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 139856, 'cpu_share': 0.0, 'mem_res': 113596}, 'contrail-schema': {'mem_virt': 89560, 'cpu_share': 0.0, 'mem_res': 80084}, 'contrail-svc-monitor': {'mem_virt': 88628, 'cpu_share': 0.0, 'mem_res': 79076}, 'contrail-device-manager': {'mem_virt': 191180, 'cpu_share': 0.0, 'mem_res': 177128}, 'contrail-config-nodemgr': {'mem_virt': 45788, 'cpu_share': 0.01, 'mem_res': 39168}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909187000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'system_cpu_usage': {'one_min_avg': 1.42, 'five_min_avg': 2.02, 'fifteen_min_avg': 2.11, 'cpu_share': 6.25, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175632, 'used': 10317464, 'free': 363332, 'buffers': 94208, 'cached': 5400628, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'ContrailConfig': {'__T': 1754909358195550, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.56"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-1."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-1."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 16855184944691956189, "uuid_lslong": 12031691376148785163}, "enable": true, "created": "2025-08-11T10:47:02.142790", "last_modified": "2025-08-11T10:47:02.142790", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"e9e9a567-93e0-49dd-a6f9-278f5a7aa80b"'}}}
__cause__ = None
__class__ =
__context__ = None
__delattr__ =
__dict__ = {}
__dir__ =
__doc__ = 'Assertion failed.'
__eq__ =
__format__ =
__ge__ =
__getattribute__ =
__gt__ =
__hash__ =
__init__ =
__init_subclass__ =
__le__ =
__lt__ =
__ne__ =
__new__ =
__reduce__ =
__reduce_ex__ =
__repr__ =
__setattr__ =
__setstate__ =
__sizeof__ =
__str__ =
__subclasshook__ =
__suppress_context__ = False
__traceback__ =
args = ('contrail-api is not connected to config db\n{\'Nod...uid\': \'"e9e9a567-93e0-49dd-a6f9-278f5a7aa80b"\'}}}',)
with_traceback =
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
result = function(self, *args, **kwargs)
File "/contrail-test/scripts/analytics/test_analytics.py", line 335, in test_verify_process_status_config
self.analytics_obj.verify_process_and_connection_infos_config()
File "/contrail-test/tcutils/collector/analytics_tests.py", line 4178, in verify_process_and_connection_infos_config
assert result, 'contrail-api is not connected to config db\n' + str(ops_inspect)
AssertionError: contrail-api is not connected to config db
{'NodeStatus': {'process_status': [{'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up'}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-device-manager', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer Connection State updated'}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754916280064757, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 24502700, 'partition_space_available_1k': 56587784, 'percentage_partition_space_used': 30}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 139856, 'cpu_share': 0.0, 'mem_res': 113596}, 'contrail-schema': {'mem_virt': 89560, 'cpu_share': 0.0, 'mem_res': 80084}, 'contrail-svc-monitor': {'mem_virt': 88628, 'cpu_share': 0.0, 'mem_res': 79076}, 'contrail-device-manager': {'mem_virt': 191180, 'cpu_share': 0.0, 'mem_res': 177128}, 'contrail-config-nodemgr': {'mem_virt': 45788, 'cpu_share': 0.01, 'mem_res': 39168}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909187000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'system_cpu_usage': {'one_min_avg': 1.42, 'five_min_avg': 2.02, 'fifteen_min_avg': 2.11, 'cpu_share': 6.25, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175632, 'used': 10317464, 'free': 363332, 'buffers': 94208, 'cached': 5400628, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'ContrailConfig': {'__T': 1754909358195550, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.56"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-1."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-1."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 16855184944691956189, "uuid_lslong": 12031691376148785163}, "enable": true, "created": "2025-08-11T10:47:02.142790", "last_modified": "2025-08-11T10:47:02.142790", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"e9e9a567-93e0-49dd-a6f9-278f5a7aa80b"'}}}
2025-08-11 12:45:22,653 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:45:22,660 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:45:22,667 - DEBUG - No XMPP flaps were noticed during the test
2025-08-11 12:45:22,667 - INFO -
2025-08-11 12:45:22,667 - INFO - END TEST : test_verify_process_status_config : FAILED[0:00:02]
2025-08-11 12:45:22,667 - INFO - --------------------------------------------------------------------------------
2025-08-11 12:58:45,261 - DEBUG - [10.0.0.56]: Running cmd : hostname
2025-08-11 12:58:45,443 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1
2025-08-11 12:58:45,444 - DEBUG - [10.0.0.56]: Running cmd : hostname -f
2025-08-11 12:58:45,503 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 12:58:45,503 - DEBUG - [10.0.0.56]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:58:45,596 - DEBUG - Output : NAMES
contrail_test_izHBaL8iO
analytics_snmp_nodemgr_1
analytics_snmp_topology_1
analytics_snmp_snmp-collector_1
analytics_alarm_alarm-gen_1
analytics_alarm_nodemgr_1
analytics_alarm_kafka_1
analytics_nodemgr_1
analytics_collector_1
analytics_api_1
analytics_database_query-engine_1
analytics_database_nodemgr_1
analytics_database_cassandra_1
control_named_1
control_control_1
control_nodemgr_1
control_dns_1
webui_job_1
webui_web_1
config_devicemgr_1
config_api_1
config_schema_1
config_dnsmasq_1
config_nodemgr_1
config_svcmonitor_1
config_database_rabbitmq_1
config_database_nodemgr_1
config_database_zookeeper_1
config_database_cassandra_1
redis_redis_1
barbican_worker
barbican_keystone_listener
barbican_api
horizon
heat_engine
heat_api_cfn
heat_api
neutron_server
nova_novncproxy
nova_conductor
nova_api
nova_scheduler
glance_api
keystone
keystone_fernet
keystone_ssh
rabbitmq
memcached
mariadb
cron
kolla_toolbox
fluentd
2025-08-11 12:58:45,597 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.0.0.56 | head -n 1 | awk '{print $2}'
2025-08-11 12:58:45,663 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 12:58:45,663 - DEBUG - [10.0.0.56]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-08-11 12:58:45,722 - DEBUG - Output : 127.0.0.1/8
10.0.0.56/24
10.20.0.17/24
172.17.0.1/16
2025-08-11 12:58:45,723 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.20.0.17 | head -n 1 | awk '{print $2}'
2025-08-11 12:58:45,782 - DEBUG - Output :
2025-08-11 12:58:45,783 - DEBUG - [10.0.0.38]: Running cmd : hostname
2025-08-11 12:58:45,954 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 12:58:45,955 - DEBUG - [10.0.0.38]: Running cmd : hostname -f
2025-08-11 12:58:46,014 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 12:58:46,015 - DEBUG - [10.0.0.38]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:58:46,103 - DEBUG - Output : NAMES
analytics_snmp_snmp-collector_1
analytics_snmp_nodemgr_1
analytics_snmp_topology_1
analytics_alarm_alarm-gen_1
analytics_alarm_nodemgr_1
analytics_alarm_kafka_1
analytics_nodemgr_1
analytics_api_1
analytics_collector_1
analytics_database_nodemgr_1
analytics_database_query-engine_1
analytics_database_cassandra_1
control_named_1
control_dns_1
control_nodemgr_1
control_control_1
webui_web_1
webui_job_1
config_devicemgr_1
config_api_1
config_dnsmasq_1
config_schema_1
config_svcmonitor_1
config_nodemgr_1
config_database_nodemgr_1
config_database_rabbitmq_1
config_database_cassandra_1
config_database_zookeeper_1
redis_redis_1
2025-08-11 12:58:46,104 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-08-11 12:58:46,162 - DEBUG - Output : 127.0.0.1/8
10.0.0.38/24
10.20.0.254/24
172.17.0.1/16
2025-08-11 12:58:46,163 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.0.0.38 | head -n 1 | awk '{print $2}'
2025-08-11 12:58:46,236 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 12:58:46,236 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.20.0.254 | head -n 1 | awk '{print $2}'
2025-08-11 12:58:46,299 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 12:58:46,300 - DEBUG - [10.0.0.22]: Running cmd : hostname
2025-08-11 12:58:46,470 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3
2025-08-11 12:58:46,471 - DEBUG - [10.0.0.22]: Running cmd : hostname -f
2025-08-11 12:58:46,528 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3.
2025-08-11 12:58:46,529 - DEBUG - [10.0.0.22]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:58:46,608 - DEBUG - Output : NAMES
analytics_snmp_snmp-collector_1
analytics_snmp_topology_1
analytics_snmp_nodemgr_1
analytics_alarm_nodemgr_1
analytics_alarm_alarm-gen_1
analytics_alarm_kafka_1
analytics_nodemgr_1
analytics_api_1
analytics_collector_1
analytics_database_query-engine_1
analytics_database_nodemgr_1
analytics_database_cassandra_1
control_named_1
control_nodemgr_1
control_dns_1
control_control_1
webui_web_1
webui_job_1
config_devicemgr_1
config_nodemgr_1
config_svcmonitor_1
config_api_1
config_schema_1
config_dnsmasq_1
config_database_rabbitmq_1
config_database_nodemgr_1
config_database_zookeeper_1
config_database_cassandra_1
redis_redis_1
2025-08-11 12:58:46,608 - DEBUG - [10.0.0.22]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-08-11 12:58:46,682 - DEBUG - Output : 127.0.0.1/8
10.0.0.22/24
10.20.0.25/24
172.17.0.1/16
2025-08-11 12:58:46,682 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.0.0.22 | head -n 1 | awk '{print $2}'
2025-08-11 12:58:46,740 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3.
2025-08-11 12:58:46,740 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.20.0.25 | head -n 1 | awk '{print $2}'
2025-08-11 12:58:46,800 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3
2025-08-11 12:58:46,801 - DEBUG - [10.0.0.19]: Running cmd : hostname
2025-08-11 12:58:46,975 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1
2025-08-11 12:58:46,975 - DEBUG - [10.0.0.19]: Running cmd : hostname -f
2025-08-11 12:58:47,033 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 12:58:47,033 - DEBUG - [10.0.0.19]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:58:47,133 - DEBUG - Output : NAMES
vrouter_vrouter-agent_1
vrouter_nodemgr_1
rsyslogd_rsyslogd_1
nova_compute
nova_libvirt
nova_ssh
cron
kolla_toolbox
fluentd
2025-08-11 12:58:47,134 - DEBUG - [10.0.0.19]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-08-11 12:58:47,193 - DEBUG - Output : 10.20.0.193/24
2025-08-11 12:58:47,193 - DEBUG - [10.0.0.19]: Running cmd : getent hosts 10.20.0.193 | head -n 1 | awk '{print $2}'
2025-08-11 12:58:47,267 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1
2025-08-11 12:58:47,268 - DEBUG - [10.0.0.41]: Running cmd : hostname
2025-08-11 12:58:47,451 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 12:58:47,451 - DEBUG - [10.0.0.41]: Running cmd : hostname -f
2025-08-11 12:58:47,515 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 12:58:47,515 - DEBUG - [10.0.0.41]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 12:58:47,614 - DEBUG - Output : NAMES
vrouter_vrouter-agent_1
vrouter_nodemgr_1
rsyslogd_rsyslogd_1
nova_compute
nova_libvirt
nova_ssh
cron
kolla_toolbox
fluentd
2025-08-11 12:58:47,614 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-08-11 12:58:47,683 - DEBUG - Output : 10.20.0.129/24
2025-08-11 12:58:47,683 - DEBUG - [10.0.0.41]: Running cmd : getent hosts 10.20.0.129 | head -n 1 | awk '{print $2}'
2025-08-11 12:58:47,758 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2
2025-08-11 12:58:48,567 - DEBUG - Not creating keypair since it exists
2025-08-11 12:58:48,941 - INFO - Using existing project ['default-domain', 'admin'](74e57a64-053a-4ab9-9a41-98124cb3c978)
2025-08-11 12:58:49,108 - INFO - ================================================================================
2025-08-11 12:58:49,108 - INFO - STARTING TEST : test_verify_process_status_config
2025-08-11 12:58:49,109 - INFO - TEST DESCRIPTION : Test to validate process_status-Config
2025-08-11 12:58:49,725 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:58:49,732 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:58:49,738 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.25': '1', '10.20.0.254': '1'}, '10.0.0.41': {'10.20.0.17': '1', '10.20.0.25': '1'}} with
2025-08-11 12:58:49,738 - INFO - Initial checks done. Running the testcase now
2025-08-11 12:58:49,738 - INFO -
2025-08-11 12:58:49,738 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 12:58:50,025 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 12:58:50,087 - ERROR - cn-jenkins-deploy-platform-ansible-os-3803-1.
2025-08-11 12:58:50,088 - ERROR - {'NodeStatus': {'process_status': [{'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up'}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-device-manager', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer Connection State updated'}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754917120156265, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 24625404, 'partition_space_available_1k': 56465080, 'percentage_partition_space_used': 30}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 106312, 'cpu_share': 0.02, 'mem_res': 93828}, 'contrail-schema': {'mem_virt': 89552, 'cpu_share': 0.01, 'mem_res': 80084}, 'contrail-svc-monitor': {'mem_virt': 88596, 'cpu_share': 0.0, 'mem_res': 79076}, 'contrail-device-manager': {'mem_virt': 191280, 'cpu_share': 0.0, 'mem_res': 177152}, 'contrail-config-nodemgr': {'mem_virt': 45812, 'cpu_share': 0.01, 'mem_res': 39168}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909187000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'system_cpu_usage': {'one_min_avg': 2.0, 'five_min_avg': 2.08, 'fifteen_min_avg': 2.08, 'cpu_share': 9.5, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175632, 'used': 10342064, 'free': 463708, 'buffers': 94904, 'cached': 5274956, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'UVEAlarms': {'__T': 1754916470751207, 'alarms': [{'type': 'default-global-system-config:system-defined-conf-incorrect', 'alarm_rules': {'or_list': [{'and_list': [{'condition': {'operation': '==', 'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'variables': []}, 'match': [{'json_operand1_value': 'null', 'json_variables': {}}]}]}]}, 'ack': False, 'timestamp': 1754916470751037, 'severity': 1, 'token': 'eyJob3N0X2lwIjogIjEwLjAuMC4zOCIsICJodHRwX3BvcnQiOiA1OTk1LCAidGltZXN0YW1wIjogMTc1NDkxNjQ3MDc1MTAzN30=', 'description': 'ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.'}]}}
2025-08-11 12:58:50,088 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat
2025-08-11 12:58:50,144 - ERROR - cn-jenkins-deploy-platform-ansible-os-3803-2.
2025-08-11 12:58:50,144 - ERROR - {'NodeStatus': {'process_status': [{'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754917122313146, 'system_mem_usage': {'total': 16175640, 'used': 6932652, 'free': 246908, 'buffers': 287148, 'cached': 8708932, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9318784, 'partition_space_available_1k': 71771700, 'percentage_partition_space_used': 11}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 99844, 'cpu_share': 0.02, 'mem_res': 96880}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 102808, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 52168, 'cpu_share': 0.01, 'mem_res': 40192}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 2.84, 'five_min_avg': 1.75, 'fifteen_min_avg': 1.47, 'cpu_share': 7.5, 'node_type': 'config-node'}, 'deleted': False}, 'UVEAlarms': {'__T': 1754916470740432, 'alarms': [{'type': 'default-global-system-config:system-defined-conf-incorrect', 'alarm_rules': {'or_list': [{'and_list': [{'condition': {'operation': '==', 'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'variables': []}, 'match': [{'json_operand1_value': 'null', 'json_variables': {}}]}]}]}, 'ack': False, 'timestamp': 1754916470740255, 'severity': 1, 'token': 'eyJob3N0X2lwIjogIjEwLjAuMC4zOCIsICJodHRwX3BvcnQiOiA1OTk1LCAidGltZXN0YW1wIjogMTc1NDkxNjQ3MDc0MDI1NX0=', 'description': 'ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.'}]}}
2025-08-11 12:58:50,144 - WARNING - No object found for module contrail-device-manager
2025-08-11 12:58:50,144 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat
2025-08-11 12:58:50,202 - ERROR - cn-jenkins-deploy-platform-ansible-os-3803-3.
2025-08-11 12:58:50,203 - ERROR - {'NodeStatus': {'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9341772, 'partition_space_available_1k': 71748712, 'percentage_partition_space_used': 12}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 102624, 'cpu_share': 0.02, 'mem_res': 98312}, 'contrail-schema': {'mem_virt': 77756, 'cpu_share': 0.0, 'mem_res': 74824}, 'contrail-svc-monitor': {'mem_virt': 75776, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 102756, 'cpu_share': 0.0, 'mem_res': 84820}, 'contrail-config-nodemgr': {'mem_virt': 52248, 'cpu_share': 0.01, 'mem_res': 40196}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, '__T': 1754917120370930, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'process_status': [{'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], 'system_cpu_usage': {'one_min_avg': 1.08, 'five_min_avg': 0.97, 'fifteen_min_avg': 0.99, 'cpu_share': 6.5, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175640, 'used': 7012288, 'free': 211008, 'buffers': 285904, 'cached': 8666440, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'UVEAlarms': {'__T': 1754916470734301, 'alarms': [{'type': 'default-global-system-config:system-defined-conf-incorrect', 'alarm_rules': {'or_list': [{'and_list': [{'condition': {'operation': '==', 'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'variables': []}, 'match': [{'json_operand1_value': 'null', 'json_variables': {}}]}]}]}, 'ack': False, 'timestamp': 1754916470734106, 'severity': 1, 'token': 'eyJob3N0X2lwIjogIjEwLjAuMC4zOCIsICJodHRwX3BvcnQiOiA1OTk1LCAidGltZXN0YW1wIjogMTc1NDkxNjQ3MDczNDEwNn0=', 'description': 'ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.'}]}}
2025-08-11 12:58:50,203 - WARNING - No object found for module contrail-device-manager
2025-08-11 12:58:50,203 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 12:58:50,228 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.56:8086 servers NOT UP
2025-08-11 12:58:50,228 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.38:8086 servers NOT UP
2025-08-11 12:58:50,228 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.22:8086 servers UP
2025-08-11 12:58:50,229 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.56:2181 servers UP
2025-08-11 12:58:50,229 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.56:9041 servers UP
2025-08-11 12:58:50,229 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.56:5673 servers UP
2025-08-11 12:58:50,229 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat
2025-08-11 12:58:50,250 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-api module connection to 10.0.0.56:8086 servers UP
2025-08-11 12:58:50,250 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-api module connection to 10.0.0.56:2181 servers UP
2025-08-11 12:58:50,251 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-api module connection to 10.0.0.56:9041 servers UP
2025-08-11 12:58:50,251 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-api module connection to 10.0.0.56:5673 servers UP
2025-08-11 12:58:50,251 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat
2025-08-11 12:58:50,274 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-api module connection to 10.0.0.56:8086 servers UP
2025-08-11 12:58:50,274 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-api module connection to 10.0.0.56:2181 servers UP
2025-08-11 12:58:50,275 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-api module connection to 10.0.0.56:9041 servers UP
2025-08-11 12:58:50,275 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-api module connection to 10.0.0.56:5673 servers UP
2025-08-11 12:58:50,275 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat
2025-08-11 12:58:50,299 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-device-manager module connection to 10.0.0.56:8086 servers NOT UP
2025-08-11 12:58:50,300 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-device-manager module connection to 10.0.0.38:8086 servers UP
2025-08-11 12:58:50,300 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-device-manager module connection to 10.0.0.56:2181 servers UP
2025-08-11 12:58:50,300 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-device-manager module connection to 10.0.0.56:9041 servers UP
2025-08-11 12:58:50,941 - ERROR - AssertionError
Python 3.6.8: /usr/bin/python3
Mon Aug 11 12:58:50 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={})
78 log.info('Initial checks done. Running the testcase now')
79 log.info('')
80 result = function(self, *args, **kwargs)
81 if self.inputs.upgrade:
82 pid = os.getpid()
result = None
function =
self =
args = ()
kwargs = {}
/contrail-test/scripts/analytics/test_analytics.py in test_verify_process_status_config(self=)
333
334 '''
335 self.analytics_obj.verify_process_and_connection_infos_config()
336
337 @test.attr(type=['cb_sanity', 'sanity', 'dev_sanity_dpdk'])
self =
self.analytics_obj =
self.analytics_obj.verify_process_and_connection_infos_config = >
/contrail-test/tcutils/collector/analytics_tests.py in verify_process_and_connection_infos_config(self=)
4235 result_cassandra = result_cassandra or self.verify_connection_infos(
4236 ops_inspect, 'contrail-device-manager', server, node=cfgm)
4237 assert result, 'contrail-device-manager is not connected to config db\n' + str(ops_inspect)
4238
4239 result = False
result = False
builtinstr =
ops_inspect = {'NodeStatus': {'process_status': [{'module_id':...fmap as ContrailConfig is missing/incorrect.'}]}}
AssertionError: contrail-device-manager is not connected to config db
{'NodeStatus': {'process_status': [{'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up'}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-device-manager', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer Connection State updated'}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754917120156265, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 24625404, 'partition_space_available_1k': 56465080, 'percentage_partition_space_used': 30}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 106312, 'cpu_share': 0.02, 'mem_res': 93828}, 'contrail-schema': {'mem_virt': 89552, 'cpu_share': 0.01, 'mem_res': 80084}, 'contrail-svc-monitor': {'mem_virt': 88596, 'cpu_share': 0.0, 'mem_res': 79076}, 'contrail-device-manager': {'mem_virt': 191280, 'cpu_share': 0.0, 'mem_res': 177152}, 'contrail-config-nodemgr': {'mem_virt': 45812, 'cpu_share': 0.01, 'mem_res': 39168}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909187000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'system_cpu_usage': {'one_min_avg': 2.0, 'five_min_avg': 2.08, 'fifteen_min_avg': 2.08, 'cpu_share': 9.5, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175632, 'used': 10342064, 'free': 463708, 'buffers': 94904, 'cached': 5274956, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'UVEAlarms': {'__T': 1754916470751207, 'alarms': [{'type': 'default-global-system-config:system-defined-conf-incorrect', 'alarm_rules': {'or_list': [{'and_list': [{'condition': {'operation': '==', 'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'variables': []}, 'match': [{'json_operand1_value': 'null', 'json_variables': {}}]}]}]}, 'ack': False, 'timestamp': 1754916470751037, 'severity': 1, 'token': 'eyJob3N0X2lwIjogIjEwLjAuMC4zOCIsICJodHRwX3BvcnQiOiA1OTk1LCAidGltZXN0YW1wIjogMTc1NDkxNjQ3MDc1MTAzN30=', 'description': 'ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.'}]}}
__cause__ = None
__class__ =
__context__ = None
__delattr__ =
__dict__ = {}
__dir__ =
__doc__ = 'Assertion failed.'
__eq__ =
__format__ =
__ge__ =
__getattribute__ =
__gt__ =
__hash__ =
__init__ =
__init_subclass__ =
__le__ =
__lt__ =
__ne__ =
__new__ =
__reduce__ =
__reduce_ex__ =
__repr__ =
__setattr__ =
__setstate__ =
__sizeof__ =
__str__ =
__subclasshook__ =
__suppress_context__ = False
__traceback__ =
args = ("contrail-device-manager is not connected to conf...fmap as ContrailConfig is missing/incorrect.'}]}}",)
with_traceback =
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper
result = function(self, *args, **kwargs)
File "/contrail-test/scripts/analytics/test_analytics.py", line 335, in test_verify_process_status_config
self.analytics_obj.verify_process_and_connection_infos_config()
File "/contrail-test/tcutils/collector/analytics_tests.py", line 4237, in verify_process_and_connection_infos_config
assert result, 'contrail-device-manager is not connected to config db\n' + str(ops_inspect)
AssertionError: contrail-device-manager is not connected to config db
{'NodeStatus': {'process_status': [{'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up'}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-device-manager', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer Connection State updated'}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754917120156265, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 24625404, 'partition_space_available_1k': 56465080, 'percentage_partition_space_used': 30}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 106312, 'cpu_share': 0.02, 'mem_res': 93828}, 'contrail-schema': {'mem_virt': 89552, 'cpu_share': 0.01, 'mem_res': 80084}, 'contrail-svc-monitor': {'mem_virt': 88596, 'cpu_share': 0.0, 'mem_res': 79076}, 'contrail-device-manager': {'mem_virt': 191280, 'cpu_share': 0.0, 'mem_res': 177152}, 'contrail-config-nodemgr': {'mem_virt': 45812, 'cpu_share': 0.01, 'mem_res': 39168}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909187000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'system_cpu_usage': {'one_min_avg': 2.0, 'five_min_avg': 2.08, 'fifteen_min_avg': 2.08, 'cpu_share': 9.5, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175632, 'used': 10342064, 'free': 463708, 'buffers': 94904, 'cached': 5274956, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'UVEAlarms': {'__T': 1754916470751207, 'alarms': [{'type': 'default-global-system-config:system-defined-conf-incorrect', 'alarm_rules': {'or_list': [{'and_list': [{'condition': {'operation': '==', 'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'variables': []}, 'match': [{'json_operand1_value': 'null', 'json_variables': {}}]}]}]}, 'ack': False, 'timestamp': 1754916470751037, 'severity': 1, 'token': 'eyJob3N0X2lwIjogIjEwLjAuMC4zOCIsICJodHRwX3BvcnQiOiA1OTk1LCAidGltZXN0YW1wIjogMTc1NDkxNjQ3MDc1MTAzN30=', 'description': 'ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.'}]}}
2025-08-11 12:58:50,941 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:58:50,948 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 12:58:50,955 - DEBUG - No XMPP flaps were noticed during the test
2025-08-11 12:58:50,955 - INFO -
2025-08-11 12:58:50,955 - INFO - END TEST : test_verify_process_status_config : FAILED[0:00:01]
2025-08-11 12:58:50,955 - INFO - --------------------------------------------------------------------------------