2025-08-11 10:52:56,011 - INFO - Using existing project ['default-domain', 'admin'](74e57a64-053a-4ab9-9a41-98124cb3c978) 2025-08-11 10:52:56,025 - INFO - ================================================================================ 2025-08-11 10:52:56,025 - INFO - STARTING TEST : test_db_nodemgr_status 2025-08-11 10:52:56,025 - INFO - TEST DESCRIPTION : Test to verify db nodemgr status 2025-08-11 10:52:56,622 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with 2025-08-11 10:52:56,622 - INFO - Initial checks done. Running the testcase now 2025-08-11 10:52:56,623 - INFO - 2025-08-11 10:52:56,623 - DEBUG - Verifying if db node_mgr running... 2025-08-11 10:52:56,623 - INFO - Verifying through collector 10.0.0.22 for db node cn-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 10:52:56,623 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/database-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 10:52:56,913 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/database-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 10:52:56,974 - INFO - Verifying through collector 10.0.0.22 for db node cn-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 10:52:56,974 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/database-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat 2025-08-11 10:52:57,031 - INFO - Verifying through collector 10.0.0.22 for db node cn-jenkins-deploy-platform-ansible-os-3803-3. 2025-08-11 10:52:57,032 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/database-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat 2025-08-11 10:52:57,691 - DEBUG - No XMPP flaps were noticed during the test 2025-08-11 10:52:57,691 - INFO - END TEST : test_db_nodemgr_status : PASSED[0:00:01] 2025-08-11 10:52:57,691 - INFO - -------------------------------------------------------------------------------- 2025-08-11 10:52:57,693 - INFO - ================================================================================ 2025-08-11 10:52:57,693 - INFO - STARTING TEST : test_verify_generator_collector_connections 2025-08-11 10:52:57,693 - INFO - TEST DESCRIPTION : Description: Verify generator:module connections to collector 1.Verify all generators connected to collector - fails otherwise 2.Get the xmpp peers in vrouter uve and get the active xmpp peer out of it 3.Verify from agent introspect that active xmpp matches with step 2 - fails otherwise 4.Get bgp peers from bgp-peer uve and verify from control node introspect that that matches - fails otherwise Maintainer: sandipd@juniper.net 2025-08-11 10:52:58,310 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with 2025-08-11 10:52:58,310 - INFO - Initial checks done. Running the testcase now 2025-08-11 10:52:58,310 - INFO - 2025-08-11 10:52:58,318 - INFO - IP 10.0.0.56 port 8083 connected to collector 10.0.0.22 2025-08-11 10:52:58,324 - INFO - IP 10.0.0.38 port 8083 connected to collector 10.0.0.22 2025-08-11 10:52:58,330 - INFO - IP 10.0.0.22 port 8083 connected to collector 10.0.0.22 2025-08-11 10:52:58,337 - INFO - IP 10.0.0.56 port 8084 connected to collector 10.0.0.22 2025-08-11 10:52:58,345 - INFO - IP 10.0.0.38 port 8084 connected to collector 10.0.0.22 2025-08-11 10:52:58,361 - INFO - IP 10.0.0.22 port 8084 connected to collector 10.0.0.38 2025-08-11 10:52:58,367 - INFO - IP 10.0.0.56 port 8087 connected to collector 10.0.0.38 2025-08-11 10:52:58,371 - INFO - IP 10.0.0.56 port 8088 connected to collector 10.0.0.56 2025-08-11 10:52:58,376 - INFO - IP 10.0.0.56 port 8090 connected to collector 10.0.0.56 2025-08-11 10:52:58,380 - INFO - IP 10.0.0.38 port 8090 connected to collector 10.0.0.56 2025-08-11 10:52:58,384 - INFO - IP 10.0.0.22 port 8090 connected to collector 10.0.0.22 2025-08-11 10:52:58,390 - INFO - IP 10.0.0.56 port 8091 connected to collector 10.0.0.56 2025-08-11 10:52:58,396 - INFO - IP 10.0.0.38 port 8091 connected to collector 10.0.0.56 2025-08-11 10:52:58,402 - INFO - IP 10.0.0.22 port 8091 connected to collector 10.0.0.38 2025-08-11 10:52:58,402 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:58,403 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-control:0?flat 2025-08-11 10:52:58,699 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-control:0?flat 2025-08-11 10:52:58,789 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-control:0 is connected to collector 10.0.0.56 2025-08-11 10:52:58,789 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:58,789 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-dns:0?flat 2025-08-11 10:52:58,869 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-dns:0 is connected to collector 10.0.0.56 2025-08-11 10:52:58,869 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:58,869 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-control:0?flat 2025-08-11 10:52:58,955 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-control:0 is connected to collector 10.0.0.56 2025-08-11 10:52:58,955 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:58,955 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-dns:0?flat 2025-08-11 10:52:58,984 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-dns:0 is connected to collector 10.0.0.56 2025-08-11 10:52:58,984 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:58,984 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-control:0?flat 2025-08-11 10:52:59,031 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-control:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,031 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,031 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-dns:0?flat 2025-08-11 10:52:59,058 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-dns:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,058 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,058 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-agent:0?flat 2025-08-11 10:52:59,095 - INFO - Validated that an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-agent:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,095 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,095 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-agent:0?flat 2025-08-11 10:52:59,132 - INFO - Validated that an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-agent:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,132 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,132 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-schema:0?flat 2025-08-11 10:52:59,151 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-schema:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,151 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,151 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-svc-monitor:0?flat 2025-08-11 10:52:59,169 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-svc-monitor:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,169 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generators 2025-08-11 10:52:59,184 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,184 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-api:0?flat 2025-08-11 10:52:59,201 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-api:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,201 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,201 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Config:contrail-api:0?flat 2025-08-11 10:52:59,218 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Config:contrail-api:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,218 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,218 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Config:contrail-api:0?flat 2025-08-11 10:52:59,235 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Config:contrail-api:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,235 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generators 2025-08-11 10:52:59,249 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,249 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0?flat 2025-08-11 10:52:59,266 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,266 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,266 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0?flat 2025-08-11 10:52:59,284 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,284 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,284 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0?flat 2025-08-11 10:52:59,301 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,301 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,301 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0?flat 2025-08-11 10:52:59,331 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,331 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,331 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0?flat 2025-08-11 10:52:59,359 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,359 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,359 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0?flat 2025-08-11 10:52:59,376 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,376 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,376 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0?flat 2025-08-11 10:52:59,393 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,393 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,393 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0?flat 2025-08-11 10:52:59,410 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,410 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,410 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0?flat 2025-08-11 10:52:59,426 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,426 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,426 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-collector:0?flat 2025-08-11 10:52:59,455 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-collector:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,455 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,455 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Database:contrail-query-engine:0?flat 2025-08-11 10:52:59,480 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Database:contrail-query-engine:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,481 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,481 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-collector:0?flat 2025-08-11 10:52:59,509 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-collector:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,509 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,509 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Database:contrail-query-engine:0?flat 2025-08-11 10:52:59,535 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Database:contrail-query-engine:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,535 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,535 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-collector:0?flat 2025-08-11 10:52:59,561 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-collector:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,561 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:52:59,561 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Database:contrail-query-engine:0?flat 2025-08-11 10:52:59,586 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Database:contrail-query-engine:0 is connected to collector 10.0.0.56 2025-08-11 10:52:59,586 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-agent:0?flat 2025-08-11 10:52:59,620 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3803-1?flat 2025-08-11 10:52:59,654 - INFO - Active xmpp peer in an-jenkins-deploy-platform-ansible-os-3803-1 vrouter uve is 10.20.0.25 2025-08-11 10:52:59,678 - DEBUG - Requesting: http://10.20.0.193:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 10:52:59,689 - INFO - Active xmpp peer in an-jenkins-deploy-platform-ansible-os-3803-1 agent introspect is 10.20.0.25 2025-08-11 10:52:59,689 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-agent:0?flat 2025-08-11 10:52:59,725 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3803-2?flat 2025-08-11 10:52:59,759 - INFO - Active xmpp peer in an-jenkins-deploy-platform-ansible-os-3803-2 vrouter uve is 10.20.0.254 2025-08-11 10:52:59,783 - DEBUG - Requesting: http://10.20.0.129:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 10:52:59,798 - INFO - Active xmpp peer in an-jenkins-deploy-platform-ansible-os-3803-2 agent introspect is 10.20.0.254 2025-08-11 10:52:59,798 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-agent:0?flat 2025-08-11 10:52:59,835 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3803-1?flat 2025-08-11 10:52:59,868 - INFO - Vrouter an-jenkins-deploy-platform-ansible-os-3803-1 connected to {'10.20.0.25', '10.20.0.17'} xmpp peers out of {'10.20.0.25', '10.20.0.17', '10.20.0.254'} 2025-08-11 10:52:59,868 - INFO - xmpp peer correctly displayed as ['10.20.0.17', '10.20.0.25'] for vrouter an-jenkins-deploy-platform-ansible-os-3803-1 2025-08-11 10:52:59,868 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-agent:0?flat 2025-08-11 10:52:59,902 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3803-2?flat 2025-08-11 10:52:59,934 - INFO - Vrouter an-jenkins-deploy-platform-ansible-os-3803-2 connected to {'10.20.0.25', '10.20.0.254'} xmpp peers out of {'10.20.0.25', '10.20.0.17', '10.20.0.254'} 2025-08-11 10:52:59,934 - INFO - xmpp peer correctly displayed as ['10.20.0.254', '10.20.0.25'] for vrouter an-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 10:52:59,934 - INFO - Verifying through opserver in 10.0.0.22 2025-08-11 10:52:59,935 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-1?flat 2025-08-11 10:52:59,956 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-2?flat 2025-08-11 10:52:59,975 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-3?flat 2025-08-11 10:52:59,995 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-1?flat 2025-08-11 10:53:00,014 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-2?flat 2025-08-11 10:53:00,034 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-3?flat 2025-08-11 10:53:00,052 - DEBUG - Verifying for cn-jenkins-deploy-platform-ansible-os-3803-1 bgp-router uve 2025-08-11 10:53:00,052 - INFO - Bgp peers = 2 2025-08-11 10:53:00,052 - DEBUG - Verifying for cn-jenkins-deploy-platform-ansible-os-3803-2 bgp-router uve 2025-08-11 10:53:00,052 - INFO - Bgp peers = 2 2025-08-11 10:53:00,053 - DEBUG - Verifying for cn-jenkins-deploy-platform-ansible-os-3803-3 bgp-router uve 2025-08-11 10:53:00,053 - INFO - Bgp peers = 2 2025-08-11 10:53:00,053 - INFO - Total xmpp peer connections found = 4 2025-08-11 10:53:00,670 - DEBUG - No XMPP flaps were noticed during the test 2025-08-11 10:53:00,670 - INFO - END TEST : test_verify_generator_collector_connections : PASSED[0:00:03] 2025-08-11 10:53:00,670 - INFO - -------------------------------------------------------------------------------- 2025-08-11 10:53:00,673 - INFO - ================================================================================ 2025-08-11 10:53:00,673 - INFO - STARTING TEST : test_verify_generator_connections_to_collector_node 2025-08-11 10:53:00,673 - INFO - TEST DESCRIPTION : Test to validate generator connections 2025-08-11 10:53:01,282 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with 2025-08-11 10:53:01,282 - INFO - Initial checks done. Running the testcase now 2025-08-11 10:53:01,282 - INFO - 2025-08-11 10:53:01,282 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,283 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-agent:0?flat 2025-08-11 10:53:01,326 - INFO - Validated that an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-agent:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,326 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,326 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-nodemgr:0?flat 2025-08-11 10:53:01,349 - INFO - Validated that an-jenkins-deploy-platform-ansible-os-3803-1:Compute:contrail-vrouter-nodemgr:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,349 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,349 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-agent:0?flat 2025-08-11 10:53:01,392 - INFO - Validated that an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-agent:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,392 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,392 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-nodemgr:0?flat 2025-08-11 10:53:01,411 - INFO - Validated that an-jenkins-deploy-platform-ansible-os-3803-2:Compute:contrail-vrouter-nodemgr:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,411 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,411 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-nodemgr:0?flat 2025-08-11 10:53:01,443 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-nodemgr:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,443 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,443 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-collector:0?flat 2025-08-11 10:53:01,481 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-collector:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,482 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,482 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0?flat 2025-08-11 10:53:01,515 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,515 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,515 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-nodemgr:0?flat 2025-08-11 10:53:01,536 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-nodemgr:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,536 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,536 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-collector:0?flat 2025-08-11 10:53:01,575 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-collector:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,575 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,575 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0?flat 2025-08-11 10:53:01,599 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,599 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,599 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-nodemgr:0?flat 2025-08-11 10:53:01,619 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-nodemgr:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,619 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,619 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-collector:0?flat 2025-08-11 10:53:01,655 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-collector:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,655 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,655 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0?flat 2025-08-11 10:53:01,676 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Analytics:contrail-analytics-api:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,677 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,677 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Database:contrail-query-engine:0?flat 2025-08-11 10:53:01,705 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Database:contrail-query-engine:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,705 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,705 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Database:contrail-database-nodemgr:0?flat 2025-08-11 10:53:01,723 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Database:contrail-database-nodemgr:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,723 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,723 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Database:contrail-query-engine:0?flat 2025-08-11 10:53:01,752 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Database:contrail-query-engine:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,753 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,753 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Database:contrail-database-nodemgr:0?flat 2025-08-11 10:53:01,775 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Database:contrail-database-nodemgr:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,776 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,776 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Database:contrail-query-engine:0?flat 2025-08-11 10:53:01,810 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Database:contrail-query-engine:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,810 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,810 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Database:contrail-database-nodemgr:0?flat 2025-08-11 10:53:01,835 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Database:contrail-database-nodemgr:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,835 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,835 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-api:0?flat 2025-08-11 10:53:01,857 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1.:Config:contrail-api:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,857 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,857 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2.:Config:contrail-api:0?flat 2025-08-11 10:53:01,880 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2.:Config:contrail-api:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,880 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,880 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3.:Config:contrail-api:0?flat 2025-08-11 10:53:01,901 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3.:Config:contrail-api:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,901 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,901 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-control:0?flat 2025-08-11 10:53:01,949 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-control:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,949 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,949 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-control-nodemgr:0?flat 2025-08-11 10:53:01,970 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-control-nodemgr:0 is connected to collector 10.0.0.56 2025-08-11 10:53:01,970 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:01,970 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-dns:0?flat 2025-08-11 10:53:02,000 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-1:Control:contrail-dns:0 is connected to collector 10.0.0.56 2025-08-11 10:53:02,000 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:02,001 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-control:0?flat 2025-08-11 10:53:02,052 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-control:0 is connected to collector 10.0.0.56 2025-08-11 10:53:02,052 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:02,052 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-control-nodemgr:0?flat 2025-08-11 10:53:02,071 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-control-nodemgr:0 is connected to collector 10.0.0.56 2025-08-11 10:53:02,071 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:02,071 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-dns:0?flat 2025-08-11 10:53:02,105 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-2:Control:contrail-dns:0 is connected to collector 10.0.0.56 2025-08-11 10:53:02,105 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:02,105 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-control:0?flat 2025-08-11 10:53:02,163 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-control:0 is connected to collector 10.0.0.56 2025-08-11 10:53:02,163 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:02,163 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-control-nodemgr:0?flat 2025-08-11 10:53:02,183 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-control-nodemgr:0 is connected to collector 10.0.0.56 2025-08-11 10:53:02,183 - DEBUG - Verifying through opserver in 10.0.0.56 2025-08-11 10:53:02,183 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/generator/cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-dns:0?flat 2025-08-11 10:53:02,216 - INFO - Validated that cn-jenkins-deploy-platform-ansible-os-3803-3:Control:contrail-dns:0 is connected to collector 10.0.0.56 2025-08-11 10:53:02,809 - DEBUG - No XMPP flaps were noticed during the test 2025-08-11 10:53:02,810 - INFO - END TEST : test_verify_generator_connections_to_collector_node : PASSED[0:00:02] 2025-08-11 10:53:02,810 - INFO - -------------------------------------------------------------------------------- 2025-08-11 10:53:02,813 - INFO - ================================================================================ 2025-08-11 10:53:02,813 - INFO - STARTING TEST : test_verify_process_status_agent 2025-08-11 10:53:02,813 - INFO - TEST DESCRIPTION : Test to validate process_status 2025-08-11 10:53:03,417 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with 2025-08-11 10:53:03,417 - INFO - Initial checks done. Running the testcase now 2025-08-11 10:53:03,417 - INFO - 2025-08-11 10:53:03,418 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/vrouter/an-jenkins-deploy-platform-ansible-os-3803-1?flat 2025-08-11 10:53:03,454 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.20.0.17:5269'] servers UP 2025-08-11 10:53:03,454 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1 connected to xmpp 10.20.0.17 2025-08-11 10:53:03,455 - WARNING - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.20.0.254:5269'] servers NOT UP 2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.20.0.25:5269'] servers UP 2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1 connected to xmpp 10.20.0.25 2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.20.0.17:53'] servers UP 2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1 connected to dns 10.20.0.17 2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.20.0.254:53'] servers UP 2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1 connected to dns 10.20.0.254 2025-08-11 10:53:03,455 - WARNING - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.20.0.25:53'] servers NOT UP 2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.0.0.56:8086'] servers UP 2025-08-11 10:53:03,455 - INFO - an-jenkins-deploy-platform-ansible-os-3803-1 connected to collector 10.0.0.56 2025-08-11 10:53:03,455 - WARNING - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.0.0.38:8086'] servers NOT UP 2025-08-11 10:53:03,456 - WARNING - an-jenkins-deploy-platform-ansible-os-3803-1:contrail-vrouter-agent module connection to ['10.0.0.22:8086'] servers NOT UP 2025-08-11 10:53:04,070 - DEBUG - No XMPP flaps were noticed during the test 2025-08-11 10:53:04,070 - INFO - END TEST : test_verify_process_status_agent : PASSED[0:00:02] 2025-08-11 10:53:04,070 - INFO - -------------------------------------------------------------------------------- 2025-08-11 10:53:04,073 - INFO - ================================================================================ 2025-08-11 10:53:04,073 - INFO - STARTING TEST : test_verify_process_status_analytics_node 2025-08-11 10:53:04,073 - INFO - TEST DESCRIPTION : Test to validate process_status-Analytics-Node 2025-08-11 10:53:04,691 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with 2025-08-11 10:53:04,691 - INFO - Initial checks done. Running the testcase now 2025-08-11 10:53:04,691 - INFO - 2025-08-11 10:53:04,691 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 10:53:04,776 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat 2025-08-11 10:53:04,814 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat 2025-08-11 10:53:04,867 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 10:53:04,911 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-collector module connection to ['10.0.0.56:8086'] servers UP 2025-08-11 10:53:04,912 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-collector module connection to ['10.0.0.56:6379'] servers NOT UP 2025-08-11 10:53:04,912 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-collector module connection to ['10.0.0.38:6379'] servers NOT UP 2025-08-11 10:53:04,912 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-collector module connection to ['10.0.0.22:6379'] servers NOT UP 2025-08-11 10:53:04,912 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-collector module connection to ['127.0.0.1:6379'] servers UP 2025-08-11 10:53:04,912 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-collector module connection to ['10.0.0.56:9042'] servers UP 2025-08-11 10:53:04,912 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat 2025-08-11 10:53:04,950 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-collector module connection to ['10.0.0.56:8086'] servers NOT UP 2025-08-11 10:53:04,950 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-collector module connection to ['10.0.0.38:8086'] servers UP 2025-08-11 10:53:04,950 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-collector module connection to ['10.0.0.56:6379'] servers NOT UP 2025-08-11 10:53:04,950 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-collector module connection to ['10.0.0.38:6379'] servers NOT UP 2025-08-11 10:53:04,950 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-collector module connection to ['10.0.0.22:6379'] servers NOT UP 2025-08-11 10:53:04,950 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-collector module connection to ['127.0.0.1:6379'] servers UP 2025-08-11 10:53:04,950 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-collector module connection to ['10.0.0.56:9042'] servers UP 2025-08-11 10:53:04,950 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat 2025-08-11 10:53:04,985 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['10.0.0.56:8086'] servers NOT UP 2025-08-11 10:53:04,985 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['10.0.0.38:8086'] servers NOT UP 2025-08-11 10:53:04,985 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['10.0.0.22:8086'] servers UP 2025-08-11 10:53:04,985 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['10.0.0.56:6379'] servers NOT UP 2025-08-11 10:53:04,986 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['10.0.0.38:6379'] servers NOT UP 2025-08-11 10:53:04,986 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['10.0.0.22:6379'] servers NOT UP 2025-08-11 10:53:04,986 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['127.0.0.1:6379'] servers UP 2025-08-11 10:53:04,986 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-collector module connection to ['10.0.0.56:9042'] servers UP 2025-08-11 10:53:04,986 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 10:53:05,026 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-analytics-api module connection to ['10.0.0.56:8086'] servers UP 2025-08-11 10:53:05,027 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat 2025-08-11 10:53:05,059 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-analytics-api module connection to ['10.0.0.56:8086'] servers UP 2025-08-11 10:53:05,059 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat 2025-08-11 10:53:05,098 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-analytics-api module connection to ['10.0.0.56:8086'] servers NOT UP 2025-08-11 10:53:05,098 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-analytics-api module connection to ['10.0.0.38:8086'] servers NOT UP 2025-08-11 10:53:05,098 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-analytics-api module connection to ['10.0.0.22:8086'] servers UP 2025-08-11 10:53:05,098 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 10:53:05,139 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-analytics-nodemgr module connection to ['10.0.0.56:8086'] servers NOT UP 2025-08-11 10:53:05,139 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-analytics-nodemgr module connection to ['10.0.0.38:8086'] servers UP 2025-08-11 10:53:05,139 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat 2025-08-11 10:53:05,172 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-analytics-nodemgr module connection to ['10.0.0.56:8086'] servers NOT UP 2025-08-11 10:53:05,172 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-analytics-nodemgr module connection to ['10.0.0.38:8086'] servers UP 2025-08-11 10:53:05,172 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/analytics-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat 2025-08-11 10:53:05,208 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-analytics-nodemgr module connection to ['10.0.0.56:8086'] servers NOT UP 2025-08-11 10:53:05,208 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-analytics-nodemgr module connection to ['10.0.0.38:8086'] servers NOT UP 2025-08-11 10:53:05,208 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-analytics-nodemgr module connection to ['10.0.0.22:8086'] servers UP 2025-08-11 10:53:05,812 - DEBUG - No XMPP flaps were noticed during the test 2025-08-11 10:53:05,812 - INFO - END TEST : test_verify_process_status_analytics_node : PASSED[0:00:01] 2025-08-11 10:53:05,812 - INFO - -------------------------------------------------------------------------------- 2025-08-11 10:53:05,815 - INFO - ================================================================================ 2025-08-11 10:53:05,815 - INFO - STARTING TEST : test_verify_process_status_config 2025-08-11 10:53:05,815 - INFO - TEST DESCRIPTION : Test to validate process_status-Config 2025-08-11 10:53:06,430 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with 2025-08-11 10:53:06,430 - INFO - Initial checks done. Running the testcase now 2025-08-11 10:53:06,430 - INFO - 2025-08-11 10:53:06,430 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 10:53:06,465 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat 2025-08-11 10:53:06,490 - WARNING - No object found for module contrail-device-manager 2025-08-11 10:53:07,113 - ERROR - AssertionError Python 3.6.8: /usr/bin/python3 Mon Aug 11 10:53:06 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/scripts/analytics/test_analytics.py in test_verify_process_status_config(self=) 333 334 ''' 335 self.analytics_obj.verify_process_and_connection_infos_config() 336 337 @test.attr(type=['cb_sanity', 'sanity', 'dev_sanity_dpdk']) self = self.analytics_obj = self.analytics_obj.verify_process_and_connection_infos_config = > /contrail-test/tcutils/collector/analytics_tests.py in verify_process_and_connection_infos_config(self=) 4126 for k in module_connection_dict: 4127 result1 = self.verify_process_status(ops_inspect, k) 4128 assert result1, f'status is invalid for {k}\n' + str(ops_inspect) 4129 4130 cfgm = self.inputs.cfgm_names[0] result1 = False builtinstr = ops_inspect = {'NodeStatus': {'process_status': [{'module_id':...uid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}} AssertionError: status is invalid for contrail-device-manager {'NodeStatus': {'process_status': [{'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754909561383898, 'system_mem_usage': {'total': 16175640, 'used': 5809908, 'free': 1998564, 'buffers': 286068, 'cached': 8081100, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 8358584, 'partition_space_available_1k': 72731900, 'percentage_partition_space_used': 10}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 98824, 'cpu_share': 0.0, 'mem_res': 94496}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 103144, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 51992, 'cpu_share': 0.01, 'mem_res': 40096}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.33, 'five_min_avg': 1.48, 'fifteen_min_avg': 0.87, 'cpu_share': 6.75, 'node_type': 'config-node'}, 'deleted': False}, 'ContrailConfig': {'__T': 1754909358195620, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.38"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-2."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-2."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1556493265386885059, "uuid_lslong": 13820473496784322762}, "enable": true, "created": "2025-08-11T10:47:02.163858", "last_modified": "2025-08-11T10:47:02.163858", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}} __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('status is invalid for contrail-device-manager\n{\'...uid\': \'"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"\'}}}',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/scripts/analytics/test_analytics.py", line 335, in test_verify_process_status_config self.analytics_obj.verify_process_and_connection_infos_config() File "/contrail-test/tcutils/collector/analytics_tests.py", line 4128, in verify_process_and_connection_infos_config assert result1, f'status is invalid for {k}\n' + str(ops_inspect) AssertionError: status is invalid for contrail-device-manager {'NodeStatus': {'process_status': [{'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754909561383898, 'system_mem_usage': {'total': 16175640, 'used': 5809908, 'free': 1998564, 'buffers': 286068, 'cached': 8081100, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 8358584, 'partition_space_available_1k': 72731900, 'percentage_partition_space_used': 10}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 98824, 'cpu_share': 0.0, 'mem_res': 94496}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 103144, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 51992, 'cpu_share': 0.01, 'mem_res': 40096}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.33, 'five_min_avg': 1.48, 'fifteen_min_avg': 0.87, 'cpu_share': 6.75, 'node_type': 'config-node'}, 'deleted': False}, 'ContrailConfig': {'__T': 1754909358195620, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.38"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-2."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-2."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1556493265386885059, "uuid_lslong": 13820473496784322762}, "enable": true, "created": "2025-08-11T10:47:02.163858", "last_modified": "2025-08-11T10:47:02.163858", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}} 2025-08-11 10:53:07,126 - DEBUG - No XMPP flaps were noticed during the test 2025-08-11 10:53:07,126 - INFO - 2025-08-11 10:53:07,126 - INFO - END TEST : test_verify_process_status_config : FAILED[0:00:02] 2025-08-11 10:53:07,126 - INFO - -------------------------------------------------------------------------------- 2025-08-11 10:53:07,129 - INFO - ================================================================================ 2025-08-11 10:53:07,129 - INFO - STARTING TEST : test_verify_process_status_control_node 2025-08-11 10:53:07,129 - INFO - TEST DESCRIPTION : Test to validate process_status-Control-Node 2025-08-11 10:53:07,725 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with 2025-08-11 10:53:07,726 - INFO - Initial checks done. Running the testcase now 2025-08-11 10:53:07,726 - INFO - 2025-08-11 10:53:07,726 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-1?flat 2025-08-11 10:53:07,747 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1:contrail-control module connection to 10.0.0.56:9041 servers UP 2025-08-11 10:53:07,747 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1:contrail-control module connection to 10.0.0.56:5673 servers UP 2025-08-11 10:53:07,747 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1:contrail-control module connection to ['10.0.0.56:8086'] servers NOT UP 2025-08-11 10:53:07,747 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1:contrail-control module connection to ['10.0.0.38:8086'] servers NOT UP 2025-08-11 10:53:07,747 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1:contrail-control module connection to ['10.0.0.22:8086'] servers UP 2025-08-11 10:53:07,747 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-2?flat 2025-08-11 10:53:07,768 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2:contrail-control module connection to 10.0.0.56:9041 servers UP 2025-08-11 10:53:07,768 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2:contrail-control module connection to 10.0.0.56:5673 servers UP 2025-08-11 10:53:07,768 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-2:contrail-control module connection to ['10.0.0.56:8086'] servers NOT UP 2025-08-11 10:53:07,768 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-2:contrail-control module connection to ['10.0.0.38:8086'] servers NOT UP 2025-08-11 10:53:07,768 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2:contrail-control module connection to ['10.0.0.22:8086'] servers UP 2025-08-11 10:53:07,768 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/control-node/cn-jenkins-deploy-platform-ansible-os-3803-3?flat 2025-08-11 10:53:07,788 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3:contrail-control module connection to 10.0.0.56:9041 servers UP 2025-08-11 10:53:07,788 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3:contrail-control module connection to 10.0.0.56:5673 servers UP 2025-08-11 10:53:07,788 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3:contrail-control module connection to ['10.0.0.56:8086'] servers NOT UP 2025-08-11 10:53:07,788 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-3:contrail-control module connection to ['10.0.0.38:8086'] servers NOT UP 2025-08-11 10:53:07,788 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3:contrail-control module connection to ['10.0.0.22:8086'] servers UP 2025-08-11 10:53:08,396 - DEBUG - No XMPP flaps were noticed during the test 2025-08-11 10:53:08,396 - INFO - END TEST : test_verify_process_status_control_node : PASSED[0:00:01] 2025-08-11 10:53:08,397 - INFO - -------------------------------------------------------------------------------- 2025-08-11 12:34:43,662 - DEBUG - [10.0.0.56]: Running cmd : hostname 2025-08-11 12:34:43,859 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1 2025-08-11 12:34:43,859 - DEBUG - [10.0.0.56]: Running cmd : hostname -f 2025-08-11 12:34:43,918 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 12:34:43,918 - DEBUG - [10.0.0.56]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:34:44,007 - DEBUG - Output : NAMES contrail_test_izHBaL8iO analytics_snmp_nodemgr_1 analytics_snmp_topology_1 analytics_snmp_snmp-collector_1 analytics_alarm_alarm-gen_1 analytics_alarm_nodemgr_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_collector_1 analytics_api_1 analytics_database_query-engine_1 analytics_database_nodemgr_1 analytics_database_cassandra_1 control_named_1 control_control_1 control_nodemgr_1 control_dns_1 webui_job_1 webui_web_1 config_devicemgr_1 config_api_1 config_schema_1 config_dnsmasq_1 config_nodemgr_1 config_svcmonitor_1 config_database_rabbitmq_1 config_database_nodemgr_1 config_database_zookeeper_1 config_database_cassandra_1 redis_redis_1 barbican_worker barbican_keystone_listener barbican_api horizon heat_engine heat_api_cfn heat_api neutron_server nova_novncproxy nova_conductor nova_api nova_scheduler glance_api keystone keystone_fernet keystone_ssh rabbitmq memcached mariadb cron kolla_toolbox fluentd 2025-08-11 12:34:44,007 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.0.0.56 | head -n 1 | awk '{print $2}' 2025-08-11 12:34:44,066 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 12:34:44,067 - DEBUG - [10.0.0.56]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-11 12:34:44,127 - DEBUG - Output : 127.0.0.1/8 10.0.0.56/24 10.20.0.17/24 172.17.0.1/16 2025-08-11 12:34:44,127 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.20.0.17 | head -n 1 | awk '{print $2}' 2025-08-11 12:34:44,186 - DEBUG - Output : 2025-08-11 12:34:44,186 - DEBUG - [10.0.0.38]: Running cmd : hostname 2025-08-11 12:34:44,363 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 12:34:44,363 - DEBUG - [10.0.0.38]: Running cmd : hostname -f 2025-08-11 12:34:44,421 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 12:34:44,422 - DEBUG - [10.0.0.38]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:34:44,513 - DEBUG - Output : NAMES analytics_snmp_snmp-collector_1 analytics_snmp_nodemgr_1 analytics_snmp_topology_1 analytics_alarm_alarm-gen_1 analytics_alarm_nodemgr_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_api_1 analytics_collector_1 analytics_database_nodemgr_1 analytics_database_query-engine_1 analytics_database_cassandra_1 control_named_1 control_dns_1 control_nodemgr_1 control_control_1 webui_web_1 webui_job_1 config_devicemgr_1 config_api_1 config_dnsmasq_1 config_schema_1 config_svcmonitor_1 config_nodemgr_1 config_database_nodemgr_1 config_database_rabbitmq_1 config_database_cassandra_1 config_database_zookeeper_1 redis_redis_1 2025-08-11 12:34:44,513 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-11 12:34:44,578 - DEBUG - Output : 127.0.0.1/8 10.0.0.38/24 10.20.0.254/24 172.17.0.1/16 2025-08-11 12:34:44,578 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.0.0.38 | head -n 1 | awk '{print $2}' 2025-08-11 12:34:44,638 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 12:34:44,638 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.20.0.254 | head -n 1 | awk '{print $2}' 2025-08-11 12:34:49,713 - DEBUG - Output : 2025-08-11 12:34:49,713 - DEBUG - [10.0.0.22]: Running cmd : hostname 2025-08-11 12:34:49,884 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3 2025-08-11 12:34:49,884 - DEBUG - [10.0.0.22]: Running cmd : hostname -f 2025-08-11 12:34:49,940 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3. 2025-08-11 12:34:49,940 - DEBUG - [10.0.0.22]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:34:50,018 - DEBUG - Output : NAMES analytics_snmp_snmp-collector_1 analytics_snmp_topology_1 analytics_snmp_nodemgr_1 analytics_alarm_nodemgr_1 analytics_alarm_alarm-gen_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_api_1 analytics_collector_1 analytics_database_query-engine_1 analytics_database_nodemgr_1 analytics_database_cassandra_1 control_named_1 control_nodemgr_1 control_dns_1 control_control_1 webui_web_1 webui_job_1 config_devicemgr_1 config_nodemgr_1 config_svcmonitor_1 config_api_1 config_schema_1 config_dnsmasq_1 config_database_rabbitmq_1 config_database_nodemgr_1 config_database_zookeeper_1 config_database_cassandra_1 redis_redis_1 2025-08-11 12:34:50,018 - DEBUG - [10.0.0.22]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-11 12:34:50,081 - DEBUG - Output : 127.0.0.1/8 10.0.0.22/24 10.20.0.25/24 172.17.0.1/16 2025-08-11 12:34:50,081 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.0.0.22 | head -n 1 | awk '{print $2}' 2025-08-11 12:34:50,141 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3. 2025-08-11 12:34:50,141 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.20.0.25 | head -n 1 | awk '{print $2}' 2025-08-11 12:34:50,201 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3 2025-08-11 12:34:50,201 - DEBUG - [10.0.0.19]: Running cmd : hostname 2025-08-11 12:34:50,413 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1 2025-08-11 12:34:50,414 - DEBUG - [10.0.0.19]: Running cmd : hostname -f 2025-08-11 12:34:50,472 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 12:34:50,472 - DEBUG - [10.0.0.19]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:34:50,553 - DEBUG - Output : NAMES vrouter_vrouter-agent_1 vrouter_nodemgr_1 rsyslogd_rsyslogd_1 nova_compute nova_libvirt nova_ssh cron kolla_toolbox fluentd 2025-08-11 12:34:50,554 - DEBUG - [10.0.0.19]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-08-11 12:34:50,612 - DEBUG - Output : 10.20.0.193/24 2025-08-11 12:34:50,612 - DEBUG - [10.0.0.19]: Running cmd : getent hosts 10.20.0.193 | head -n 1 | awk '{print $2}' 2025-08-11 12:34:50,687 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1 2025-08-11 12:34:50,687 - DEBUG - [10.0.0.41]: Running cmd : hostname 2025-08-11 12:34:50,858 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 12:34:50,858 - DEBUG - [10.0.0.41]: Running cmd : hostname -f 2025-08-11 12:34:50,919 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 12:34:50,919 - DEBUG - [10.0.0.41]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:34:50,999 - DEBUG - Output : NAMES vrouter_vrouter-agent_1 vrouter_nodemgr_1 rsyslogd_rsyslogd_1 nova_compute nova_libvirt nova_ssh cron kolla_toolbox fluentd 2025-08-11 12:34:50,999 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-08-11 12:34:51,058 - DEBUG - Output : 10.20.0.129/24 2025-08-11 12:34:51,058 - DEBUG - [10.0.0.41]: Running cmd : getent hosts 10.20.0.129 | head -n 1 | awk '{print $2}' 2025-08-11 12:34:51,123 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 12:34:51,959 - DEBUG - Not creating keypair since it exists 2025-08-11 12:34:52,323 - INFO - Using existing project ['default-domain', 'admin'](74e57a64-053a-4ab9-9a41-98124cb3c978) 2025-08-11 12:34:52,494 - INFO - ================================================================================ 2025-08-11 12:34:52,494 - INFO - STARTING TEST : test_verify_process_status_config 2025-08-11 12:34:52,495 - INFO - TEST DESCRIPTION : Test to validate process_status-Config 2025-08-11 12:34:53,093 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:34:53,100 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:34:53,106 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.17': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.25': '0'}} with 2025-08-11 12:34:53,106 - INFO - Initial checks done. Running the testcase now 2025-08-11 12:34:53,106 - INFO - 2025-08-11 12:34:53,106 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 12:34:53,396 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 12:34:53,458 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat 2025-08-11 12:34:53,517 - WARNING - No object found for module contrail-device-manager 2025-08-11 12:34:54,120 - ERROR - AssertionError Python 3.6.8: /usr/bin/python3 Mon Aug 11 12:34:53 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/scripts/analytics/test_analytics.py in test_verify_process_status_config(self=) 333 334 ''' 335 self.analytics_obj.verify_process_and_connection_infos_config() 336 337 @test.attr(type=['cb_sanity', 'sanity', 'dev_sanity_dpdk']) self = self.analytics_obj = self.analytics_obj.verify_process_and_connection_infos_config = > /contrail-test/tcutils/collector/analytics_tests.py in verify_process_and_connection_infos_config(self=) 4126 for k in module_connection_dict: 4127 result1 = self.verify_process_status(ops_inspect, k) 4128 assert result1, f'status is invalid for {k}\n' + str(ops_inspect) 4129 4130 cfgm = self.inputs.cfgm_names[0] result1 = False builtinstr = ops_inspect = {'NodeStatus': {'process_status': [{'module_id':...uid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}} AssertionError: status is invalid for contrail-device-manager {'NodeStatus': {'process_status': [{'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754915682118270, 'system_mem_usage': {'total': 16175640, 'used': 6773708, 'free': 233396, 'buffers': 291184, 'cached': 8877352, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9153532, 'partition_space_available_1k': 71936952, 'percentage_partition_space_used': 11}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 134472, 'cpu_share': 0.01, 'mem_res': 109176}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 103136, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 52260, 'cpu_share': 0.01, 'mem_res': 40192}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.06, 'five_min_avg': 1.31, 'fifteen_min_avg': 1.43, 'cpu_share': 7.0, 'node_type': 'config-node'}, 'deleted': False}, 'ContrailConfig': {'__T': 1754909358195620, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.38"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-2."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-2."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1556493265386885059, "uuid_lslong": 13820473496784322762}, "enable": true, "created": "2025-08-11T10:47:02.163858", "last_modified": "2025-08-11T10:47:02.163858", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}} __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('status is invalid for contrail-device-manager\n{\'...uid\': \'"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"\'}}}',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/scripts/analytics/test_analytics.py", line 335, in test_verify_process_status_config self.analytics_obj.verify_process_and_connection_infos_config() File "/contrail-test/tcutils/collector/analytics_tests.py", line 4128, in verify_process_and_connection_infos_config assert result1, f'status is invalid for {k}\n' + str(ops_inspect) AssertionError: status is invalid for contrail-device-manager {'NodeStatus': {'process_status': [{'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754915682118270, 'system_mem_usage': {'total': 16175640, 'used': 6773708, 'free': 233396, 'buffers': 291184, 'cached': 8877352, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9153532, 'partition_space_available_1k': 71936952, 'percentage_partition_space_used': 11}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 134472, 'cpu_share': 0.01, 'mem_res': 109176}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 103136, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 52260, 'cpu_share': 0.01, 'mem_res': 40192}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.06, 'five_min_avg': 1.31, 'fifteen_min_avg': 1.43, 'cpu_share': 7.0, 'node_type': 'config-node'}, 'deleted': False}, 'ContrailConfig': {'__T': 1754909358195620, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.38"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-2."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-2."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1556493265386885059, "uuid_lslong": 13820473496784322762}, "enable": true, "created": "2025-08-11T10:47:02.163858", "last_modified": "2025-08-11T10:47:02.163858", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}} 2025-08-11 12:34:54,121 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:34:54,128 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:34:54,134 - DEBUG - No XMPP flaps were noticed during the test 2025-08-11 12:34:54,134 - INFO - 2025-08-11 12:34:54,134 - INFO - END TEST : test_verify_process_status_config : FAILED[0:00:02] 2025-08-11 12:34:54,134 - INFO - -------------------------------------------------------------------------------- 2025-08-11 12:39:02,931 - DEBUG - [10.0.0.56]: Running cmd : hostname 2025-08-11 12:39:03,123 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1 2025-08-11 12:39:03,123 - DEBUG - [10.0.0.56]: Running cmd : hostname -f 2025-08-11 12:39:03,182 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 12:39:03,182 - DEBUG - [10.0.0.56]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:39:03,327 - DEBUG - Output : NAMES contrail_test_izHBaL8iO analytics_snmp_nodemgr_1 analytics_snmp_topology_1 analytics_snmp_snmp-collector_1 analytics_alarm_alarm-gen_1 analytics_alarm_nodemgr_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_collector_1 analytics_api_1 analytics_database_query-engine_1 analytics_database_nodemgr_1 analytics_database_cassandra_1 control_named_1 control_control_1 control_nodemgr_1 control_dns_1 webui_job_1 webui_web_1 config_devicemgr_1 config_api_1 config_schema_1 config_dnsmasq_1 config_nodemgr_1 config_svcmonitor_1 config_database_rabbitmq_1 config_database_nodemgr_1 config_database_zookeeper_1 config_database_cassandra_1 redis_redis_1 barbican_worker barbican_keystone_listener barbican_api horizon heat_engine heat_api_cfn heat_api neutron_server nova_novncproxy nova_conductor nova_api nova_scheduler glance_api keystone keystone_fernet keystone_ssh rabbitmq memcached mariadb cron kolla_toolbox fluentd 2025-08-11 12:39:03,328 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.0.0.56 | head -n 1 | awk '{print $2}' 2025-08-11 12:39:03,382 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 12:39:03,383 - DEBUG - [10.0.0.56]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-11 12:39:03,442 - DEBUG - Output : 127.0.0.1/8 10.0.0.56/24 10.20.0.17/24 172.17.0.1/16 2025-08-11 12:39:03,442 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.20.0.17 | head -n 1 | awk '{print $2}' 2025-08-11 12:39:03,502 - DEBUG - Output : 2025-08-11 12:39:03,502 - DEBUG - [10.0.0.38]: Running cmd : hostname 2025-08-11 12:39:03,683 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 12:39:03,683 - DEBUG - [10.0.0.38]: Running cmd : hostname -f 2025-08-11 12:39:03,741 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 12:39:03,742 - DEBUG - [10.0.0.38]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:39:03,822 - DEBUG - Output : NAMES analytics_snmp_snmp-collector_1 analytics_snmp_nodemgr_1 analytics_snmp_topology_1 analytics_alarm_alarm-gen_1 analytics_alarm_nodemgr_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_api_1 analytics_collector_1 analytics_database_nodemgr_1 analytics_database_query-engine_1 analytics_database_cassandra_1 control_named_1 control_dns_1 control_nodemgr_1 control_control_1 webui_web_1 webui_job_1 config_devicemgr_1 config_api_1 config_dnsmasq_1 config_schema_1 config_svcmonitor_1 config_nodemgr_1 config_database_nodemgr_1 config_database_rabbitmq_1 config_database_cassandra_1 config_database_zookeeper_1 redis_redis_1 2025-08-11 12:39:03,822 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-11 12:39:03,886 - DEBUG - Output : 127.0.0.1/8 10.0.0.38/24 10.20.0.254/24 172.17.0.1/16 2025-08-11 12:39:03,886 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.0.0.38 | head -n 1 | awk '{print $2}' 2025-08-11 12:39:03,945 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 12:39:03,946 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.20.0.254 | head -n 1 | awk '{print $2}' 2025-08-11 12:39:04,097 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 12:39:04,098 - DEBUG - [10.0.0.22]: Running cmd : hostname 2025-08-11 12:39:04,278 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3 2025-08-11 12:39:04,278 - DEBUG - [10.0.0.22]: Running cmd : hostname -f 2025-08-11 12:39:04,337 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3. 2025-08-11 12:39:04,337 - DEBUG - [10.0.0.22]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:39:04,418 - DEBUG - Output : NAMES analytics_snmp_snmp-collector_1 analytics_snmp_topology_1 analytics_snmp_nodemgr_1 analytics_alarm_nodemgr_1 analytics_alarm_alarm-gen_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_api_1 analytics_collector_1 analytics_database_query-engine_1 analytics_database_nodemgr_1 analytics_database_cassandra_1 control_named_1 control_nodemgr_1 control_dns_1 control_control_1 webui_web_1 webui_job_1 config_devicemgr_1 config_nodemgr_1 config_svcmonitor_1 config_api_1 config_schema_1 config_dnsmasq_1 config_database_rabbitmq_1 config_database_nodemgr_1 config_database_zookeeper_1 config_database_cassandra_1 redis_redis_1 2025-08-11 12:39:04,418 - DEBUG - [10.0.0.22]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-11 12:39:04,481 - DEBUG - Output : 127.0.0.1/8 10.0.0.22/24 10.20.0.25/24 172.17.0.1/16 2025-08-11 12:39:04,481 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.0.0.22 | head -n 1 | awk '{print $2}' 2025-08-11 12:39:04,540 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3. 2025-08-11 12:39:04,540 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.20.0.25 | head -n 1 | awk '{print $2}' 2025-08-11 12:39:04,615 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3 2025-08-11 12:39:04,615 - DEBUG - [10.0.0.19]: Running cmd : hostname 2025-08-11 12:39:04,786 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1 2025-08-11 12:39:04,786 - DEBUG - [10.0.0.19]: Running cmd : hostname -f 2025-08-11 12:39:04,848 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 12:39:04,848 - DEBUG - [10.0.0.19]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:39:04,929 - DEBUG - Output : NAMES vrouter_vrouter-agent_1 vrouter_nodemgr_1 rsyslogd_rsyslogd_1 nova_compute nova_libvirt nova_ssh cron kolla_toolbox fluentd 2025-08-11 12:39:04,929 - DEBUG - [10.0.0.19]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-08-11 12:39:04,985 - DEBUG - Output : 10.20.0.193/24 2025-08-11 12:39:04,985 - DEBUG - [10.0.0.19]: Running cmd : getent hosts 10.20.0.193 | head -n 1 | awk '{print $2}' 2025-08-11 12:39:05,044 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1 2025-08-11 12:39:05,044 - DEBUG - [10.0.0.41]: Running cmd : hostname 2025-08-11 12:39:05,211 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 12:39:05,211 - DEBUG - [10.0.0.41]: Running cmd : hostname -f 2025-08-11 12:39:05,271 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 12:39:05,271 - DEBUG - [10.0.0.41]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:39:05,351 - DEBUG - Output : NAMES vrouter_vrouter-agent_1 vrouter_nodemgr_1 rsyslogd_rsyslogd_1 nova_compute nova_libvirt nova_ssh cron kolla_toolbox fluentd 2025-08-11 12:39:05,352 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-08-11 12:39:05,415 - DEBUG - Output : 10.20.0.129/24 2025-08-11 12:39:05,415 - DEBUG - [10.0.0.41]: Running cmd : getent hosts 10.20.0.129 | head -n 1 | awk '{print $2}' 2025-08-11 12:39:05,479 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 12:39:06,470 - DEBUG - Not creating keypair since it exists 2025-08-11 12:39:07,029 - INFO - Using existing project ['default-domain', 'admin'](74e57a64-053a-4ab9-9a41-98124cb3c978) 2025-08-11 12:39:07,317 - INFO - ================================================================================ 2025-08-11 12:39:07,317 - INFO - STARTING TEST : test_verify_process_status_config 2025-08-11 12:39:07,319 - INFO - TEST DESCRIPTION : Test to validate process_status-Config 2025-08-11 12:39:07,961 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:39:07,984 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:39:07,990 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.254': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.17': '0'}} with 2025-08-11 12:39:07,990 - INFO - Initial checks done. Running the testcase now 2025-08-11 12:39:07,990 - INFO - 2025-08-11 12:39:07,991 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 12:39:08,520 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 12:39:08,633 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat 2025-08-11 12:39:08,778 - WARNING - No object found for module contrail-device-manager 2025-08-11 12:39:09,409 - ERROR - AssertionError Python 3.6.8: /usr/bin/python3 Mon Aug 11 12:39:08 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/scripts/analytics/test_analytics.py in test_verify_process_status_config(self=) 333 334 ''' 335 self.analytics_obj.verify_process_and_connection_infos_config() 336 337 @test.attr(type=['cb_sanity', 'sanity', 'dev_sanity_dpdk']) self = self.analytics_obj = self.analytics_obj.verify_process_and_connection_infos_config = > /contrail-test/tcutils/collector/analytics_tests.py in verify_process_and_connection_infos_config(self=) 4126 for k in module_connection_dict: 4127 result1 = self.verify_process_status(ops_inspect, k) 4128 assert result1, f'status is invalid for {cfgm} in {k}\n' + str(ops_inspect) 4129 4130 cfgm = self.inputs.cfgm_names[0] result1 = False builtinstr = ops_inspect = {'NodeStatus': {'process_status': [{'module_id':...uid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}} AssertionError: status is invalid for cn-jenkins-deploy-platform-ansible-os-3803-2. in contrail-device-manager {'NodeStatus': {'process_status': [{'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754915922114140, 'system_mem_usage': {'total': 16175640, 'used': 6808980, 'free': 223664, 'buffers': 289416, 'cached': 8853580, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9171164, 'partition_space_available_1k': 71919320, 'percentage_partition_space_used': 11}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 135256, 'cpu_share': 0.02, 'mem_res': 109680}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 103112, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 52260, 'cpu_share': 0.01, 'mem_res': 40192}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.01, 'five_min_avg': 1.14, 'fifteen_min_avg': 1.33, 'cpu_share': 7.25, 'node_type': 'config-node'}, 'deleted': False}, 'ContrailConfig': {'__T': 1754909358195620, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.38"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-2."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-2."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1556493265386885059, "uuid_lslong": 13820473496784322762}, "enable": true, "created": "2025-08-11T10:47:02.163858", "last_modified": "2025-08-11T10:47:02.163858", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}} __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('status is invalid for cn-jenkins-deploy-platform...uid\': \'"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"\'}}}',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/scripts/analytics/test_analytics.py", line 335, in test_verify_process_status_config self.analytics_obj.verify_process_and_connection_infos_config() File "/contrail-test/tcutils/collector/analytics_tests.py", line 4128, in verify_process_and_connection_infos_config assert result1, f'status is invalid for {cfgm} in {k}\n' + str(ops_inspect) AssertionError: status is invalid for cn-jenkins-deploy-platform-ansible-os-3803-2. in contrail-device-manager {'NodeStatus': {'process_status': [{'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754915922114140, 'system_mem_usage': {'total': 16175640, 'used': 6808980, 'free': 223664, 'buffers': 289416, 'cached': 8853580, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9171164, 'partition_space_available_1k': 71919320, 'percentage_partition_space_used': 11}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 135256, 'cpu_share': 0.02, 'mem_res': 109680}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 103112, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 52260, 'cpu_share': 0.01, 'mem_res': 40192}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.01, 'five_min_avg': 1.14, 'fifteen_min_avg': 1.33, 'cpu_share': 7.25, 'node_type': 'config-node'}, 'deleted': False}, 'ContrailConfig': {'__T': 1754909358195620, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.38"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-2."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-2."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1556493265386885059, "uuid_lslong": 13820473496784322762}, "enable": true, "created": "2025-08-11T10:47:02.163858", "last_modified": "2025-08-11T10:47:02.163858", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}} 2025-08-11 12:39:09,409 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:39:09,416 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:39:09,422 - DEBUG - No XMPP flaps were noticed during the test 2025-08-11 12:39:09,422 - INFO - 2025-08-11 12:39:09,422 - INFO - END TEST : test_verify_process_status_config : FAILED[0:00:02] 2025-08-11 12:39:09,422 - INFO - -------------------------------------------------------------------------------- 2025-08-11 12:45:11,916 - DEBUG - [10.0.0.56]: Running cmd : hostname 2025-08-11 12:45:12,107 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1 2025-08-11 12:45:12,108 - DEBUG - [10.0.0.56]: Running cmd : hostname -f 2025-08-11 12:45:12,166 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 12:45:12,167 - DEBUG - [10.0.0.56]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:45:12,282 - DEBUG - Output : NAMES contrail_test_izHBaL8iO analytics_snmp_nodemgr_1 analytics_snmp_topology_1 analytics_snmp_snmp-collector_1 analytics_alarm_alarm-gen_1 analytics_alarm_nodemgr_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_collector_1 analytics_api_1 analytics_database_query-engine_1 analytics_database_nodemgr_1 analytics_database_cassandra_1 control_named_1 control_control_1 control_nodemgr_1 control_dns_1 webui_job_1 webui_web_1 config_devicemgr_1 config_api_1 config_schema_1 config_dnsmasq_1 config_nodemgr_1 config_svcmonitor_1 config_database_rabbitmq_1 config_database_nodemgr_1 config_database_zookeeper_1 config_database_cassandra_1 redis_redis_1 barbican_worker barbican_keystone_listener barbican_api horizon heat_engine heat_api_cfn heat_api neutron_server nova_novncproxy nova_conductor nova_api nova_scheduler glance_api keystone keystone_fernet keystone_ssh rabbitmq memcached mariadb cron kolla_toolbox fluentd 2025-08-11 12:45:12,282 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.0.0.56 | head -n 1 | awk '{print $2}' 2025-08-11 12:45:12,352 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 12:45:12,353 - DEBUG - [10.0.0.56]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-11 12:45:12,410 - DEBUG - Output : 127.0.0.1/8 10.0.0.56/24 10.20.0.17/24 172.17.0.1/16 2025-08-11 12:45:12,410 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.20.0.17 | head -n 1 | awk '{print $2}' 2025-08-11 12:45:12,471 - DEBUG - Output : 2025-08-11 12:45:12,471 - DEBUG - [10.0.0.38]: Running cmd : hostname 2025-08-11 12:45:12,654 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 12:45:12,655 - DEBUG - [10.0.0.38]: Running cmd : hostname -f 2025-08-11 12:45:12,713 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 12:45:12,714 - DEBUG - [10.0.0.38]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:45:12,805 - DEBUG - Output : NAMES analytics_snmp_snmp-collector_1 analytics_snmp_nodemgr_1 analytics_snmp_topology_1 analytics_alarm_alarm-gen_1 analytics_alarm_nodemgr_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_api_1 analytics_collector_1 analytics_database_nodemgr_1 analytics_database_query-engine_1 analytics_database_cassandra_1 control_named_1 control_dns_1 control_nodemgr_1 control_control_1 webui_web_1 webui_job_1 config_devicemgr_1 config_api_1 config_dnsmasq_1 config_schema_1 config_svcmonitor_1 config_nodemgr_1 config_database_nodemgr_1 config_database_rabbitmq_1 config_database_cassandra_1 config_database_zookeeper_1 redis_redis_1 2025-08-11 12:45:12,805 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-11 12:45:12,868 - DEBUG - Output : 127.0.0.1/8 10.0.0.38/24 10.20.0.254/24 172.17.0.1/16 2025-08-11 12:45:12,868 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.0.0.38 | head -n 1 | awk '{print $2}' 2025-08-11 12:45:12,926 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 12:45:12,927 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.20.0.254 | head -n 1 | awk '{print $2}' 2025-08-11 12:45:17,999 - DEBUG - Output : 2025-08-11 12:45:17,999 - DEBUG - [10.0.0.22]: Running cmd : hostname 2025-08-11 12:45:18,167 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3 2025-08-11 12:45:18,167 - DEBUG - [10.0.0.22]: Running cmd : hostname -f 2025-08-11 12:45:18,224 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3. 2025-08-11 12:45:18,224 - DEBUG - [10.0.0.22]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:45:18,305 - DEBUG - Output : NAMES analytics_snmp_snmp-collector_1 analytics_snmp_topology_1 analytics_snmp_nodemgr_1 analytics_alarm_nodemgr_1 analytics_alarm_alarm-gen_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_api_1 analytics_collector_1 analytics_database_query-engine_1 analytics_database_nodemgr_1 analytics_database_cassandra_1 control_named_1 control_nodemgr_1 control_dns_1 control_control_1 webui_web_1 webui_job_1 config_devicemgr_1 config_nodemgr_1 config_svcmonitor_1 config_api_1 config_schema_1 config_dnsmasq_1 config_database_rabbitmq_1 config_database_nodemgr_1 config_database_zookeeper_1 config_database_cassandra_1 redis_redis_1 2025-08-11 12:45:18,305 - DEBUG - [10.0.0.22]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-11 12:45:18,368 - DEBUG - Output : 127.0.0.1/8 10.0.0.22/24 10.20.0.25/24 172.17.0.1/16 2025-08-11 12:45:18,369 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.0.0.22 | head -n 1 | awk '{print $2}' 2025-08-11 12:45:18,432 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3. 2025-08-11 12:45:18,432 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.20.0.25 | head -n 1 | awk '{print $2}' 2025-08-11 12:45:18,497 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3 2025-08-11 12:45:18,497 - DEBUG - [10.0.0.19]: Running cmd : hostname 2025-08-11 12:45:18,915 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1 2025-08-11 12:45:18,916 - DEBUG - [10.0.0.19]: Running cmd : hostname -f 2025-08-11 12:45:18,972 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 12:45:18,972 - DEBUG - [10.0.0.19]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:45:19,053 - DEBUG - Output : NAMES vrouter_vrouter-agent_1 vrouter_nodemgr_1 rsyslogd_rsyslogd_1 nova_compute nova_libvirt nova_ssh cron kolla_toolbox fluentd 2025-08-11 12:45:19,053 - DEBUG - [10.0.0.19]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-08-11 12:45:19,109 - DEBUG - Output : 10.20.0.193/24 2025-08-11 12:45:19,109 - DEBUG - [10.0.0.19]: Running cmd : getent hosts 10.20.0.193 | head -n 1 | awk '{print $2}' 2025-08-11 12:45:19,168 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1 2025-08-11 12:45:19,168 - DEBUG - [10.0.0.41]: Running cmd : hostname 2025-08-11 12:45:19,338 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 12:45:19,338 - DEBUG - [10.0.0.41]: Running cmd : hostname -f 2025-08-11 12:45:19,400 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 12:45:19,400 - DEBUG - [10.0.0.41]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:45:19,479 - DEBUG - Output : NAMES vrouter_vrouter-agent_1 vrouter_nodemgr_1 rsyslogd_rsyslogd_1 nova_compute nova_libvirt nova_ssh cron kolla_toolbox fluentd 2025-08-11 12:45:19,479 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-08-11 12:45:19,535 - DEBUG - Output : 10.20.0.129/24 2025-08-11 12:45:19,535 - DEBUG - [10.0.0.41]: Running cmd : getent hosts 10.20.0.129 | head -n 1 | awk '{print $2}' 2025-08-11 12:45:19,595 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 12:45:20,406 - DEBUG - Not creating keypair since it exists 2025-08-11 12:45:20,787 - INFO - Using existing project ['default-domain', 'admin'](74e57a64-053a-4ab9-9a41-98124cb3c978) 2025-08-11 12:45:20,965 - INFO - ================================================================================ 2025-08-11 12:45:20,966 - INFO - STARTING TEST : test_verify_process_status_config 2025-08-11 12:45:20,966 - INFO - TEST DESCRIPTION : Test to validate process_status-Config 2025-08-11 12:45:21,557 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:45:21,564 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:45:21,570 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.254': '0', '10.20.0.25': '0'}, '10.0.0.41': {'10.20.0.254': '0', '10.20.0.17': '1'}} with 2025-08-11 12:45:21,570 - INFO - Initial checks done. Running the testcase now 2025-08-11 12:45:21,570 - INFO - 2025-08-11 12:45:21,570 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 12:45:21,859 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 12:45:21,922 - ERROR - cn-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 12:45:21,922 - ERROR - {'NodeStatus': {'process_status': [{'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up'}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-device-manager', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer Connection State updated'}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754916280064757, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 24502700, 'partition_space_available_1k': 56587784, 'percentage_partition_space_used': 30}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 139856, 'cpu_share': 0.0, 'mem_res': 113596}, 'contrail-schema': {'mem_virt': 89560, 'cpu_share': 0.0, 'mem_res': 80084}, 'contrail-svc-monitor': {'mem_virt': 88628, 'cpu_share': 0.0, 'mem_res': 79076}, 'contrail-device-manager': {'mem_virt': 191180, 'cpu_share': 0.0, 'mem_res': 177128}, 'contrail-config-nodemgr': {'mem_virt': 45788, 'cpu_share': 0.01, 'mem_res': 39168}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909187000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'system_cpu_usage': {'one_min_avg': 1.42, 'five_min_avg': 2.02, 'fifteen_min_avg': 2.11, 'cpu_share': 6.25, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175632, 'used': 10317464, 'free': 363332, 'buffers': 94208, 'cached': 5400628, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'ContrailConfig': {'__T': 1754909358195550, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.56"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-1."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-1."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 16855184944691956189, "uuid_lslong": 12031691376148785163}, "enable": true, "created": "2025-08-11T10:47:02.142790", "last_modified": "2025-08-11T10:47:02.142790", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"e9e9a567-93e0-49dd-a6f9-278f5a7aa80b"'}}} 2025-08-11 12:45:21,922 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat 2025-08-11 12:45:22,004 - ERROR - cn-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 12:45:22,004 - ERROR - {'NodeStatus': {'process_status': [{'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754916282181580, 'system_mem_usage': {'total': 16175640, 'used': 6940552, 'free': 251496, 'buffers': 287940, 'cached': 8695652, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9225172, 'partition_space_available_1k': 71865312, 'percentage_partition_space_used': 11}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 139076, 'cpu_share': 0.0, 'mem_res': 111356}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 102876, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 52168, 'cpu_share': 0.01, 'mem_res': 40192}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 1.29, 'five_min_avg': 1.4, 'fifteen_min_avg': 1.42, 'cpu_share': 6.5, 'node_type': 'config-node'}, 'deleted': False}, 'ContrailConfig': {'__T': 1754909358195620, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.38"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-2."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-2."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1556493265386885059, "uuid_lslong": 13820473496784322762}, "enable": true, "created": "2025-08-11T10:47:02.163858", "last_modified": "2025-08-11T10:47:02.163858", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"1599c660-d2ac-4fc3-bfcc-2f6bdb87c0ca"'}}} 2025-08-11 12:45:22,004 - WARNING - No object found for module contrail-device-manager 2025-08-11 12:45:22,004 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat 2025-08-11 12:45:22,031 - ERROR - cn-jenkins-deploy-platform-ansible-os-3803-3. 2025-08-11 12:45:22,031 - ERROR - {'NodeStatus': {'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9245104, 'partition_space_available_1k': 71845380, 'percentage_partition_space_used': 11}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 165268, 'cpu_share': 0.0, 'mem_res': 118396}, 'contrail-schema': {'mem_virt': 77756, 'cpu_share': 0.0, 'mem_res': 74824}, 'contrail-svc-monitor': {'mem_virt': 75776, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 102832, 'cpu_share': 0.0, 'mem_res': 84820}, 'contrail-config-nodemgr': {'mem_virt': 52232, 'cpu_share': 0.01, 'mem_res': 40188}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, '__T': 1754916280266459, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'process_status': [{'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], 'system_cpu_usage': {'one_min_avg': 0.92, 'five_min_avg': 1.08, 'fifteen_min_avg': 1.09, 'cpu_share': 5.25, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175640, 'used': 6999768, 'free': 245776, 'buffers': 285784, 'cached': 8644312, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'ContrailConfig': {'__T': 1754909358195771, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.22"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-3."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-3."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12779472725754465482, "uuid_lslong": 11944146654837029108}, "enable": true, "created": "2025-08-11T10:47:02.187565", "last_modified": "2025-08-11T10:47:02.187565", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"b159cebc-4b63-4cca-a5c2-22198abb38f4"'}}} 2025-08-11 12:45:22,031 - WARNING - No object found for module contrail-device-manager 2025-08-11 12:45:22,031 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 12:45:22,059 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.56:8086 servers NOT UP 2025-08-11 12:45:22,059 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.38:8086 servers NOT UP 2025-08-11 12:45:22,059 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.22:8086 servers UP 2025-08-11 12:45:22,059 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.56:2181 servers UP 2025-08-11 12:45:22,060 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.56:9161 servers NOT UP 2025-08-11 12:45:22,060 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.38:9161 servers NOT UP 2025-08-11 12:45:22,060 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.22:9161 servers NOT UP 2025-08-11 12:45:22,060 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to cn-jenkins-deploy-platform-ansible-os-3803-1.*:9161 servers NOT UP 2025-08-11 12:45:22,060 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to cn-jenkins-deploy-platform-ansible-os-3803-2.*:9161 servers NOT UP 2025-08-11 12:45:22,061 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to cn-jenkins-deploy-platform-ansible-os-3803-3.*:9161 servers NOT UP 2025-08-11 12:45:22,653 - ERROR - AssertionError Python 3.6.8: /usr/bin/python3 Mon Aug 11 12:45:22 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/scripts/analytics/test_analytics.py in test_verify_process_status_config(self=) 333 334 ''' 335 self.analytics_obj.verify_process_and_connection_infos_config() 336 337 @test.attr(type=['cb_sanity', 'sanity', 'dev_sanity_dpdk']) self = self.analytics_obj = self.analytics_obj.verify_process_and_connection_infos_config = > /contrail-test/tcutils/collector/analytics_tests.py in verify_process_and_connection_infos_config(self=) 4176 result = result or self.verify_connection_infos( 4177 ops_inspect, 'contrail-api', server, node=cfgm) 4178 assert result, 'contrail-api is not connected to config db\n' + str(ops_inspect) 4179 4180 result = False result = False builtinstr = ops_inspect = {'NodeStatus': {'process_status': [{'module_id':...uid': '"e9e9a567-93e0-49dd-a6f9-278f5a7aa80b"'}}} AssertionError: contrail-api is not connected to config db {'NodeStatus': {'process_status': [{'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up'}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-device-manager', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer Connection State updated'}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754916280064757, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 24502700, 'partition_space_available_1k': 56587784, 'percentage_partition_space_used': 30}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 139856, 'cpu_share': 0.0, 'mem_res': 113596}, 'contrail-schema': {'mem_virt': 89560, 'cpu_share': 0.0, 'mem_res': 80084}, 'contrail-svc-monitor': {'mem_virt': 88628, 'cpu_share': 0.0, 'mem_res': 79076}, 'contrail-device-manager': {'mem_virt': 191180, 'cpu_share': 0.0, 'mem_res': 177128}, 'contrail-config-nodemgr': {'mem_virt': 45788, 'cpu_share': 0.01, 'mem_res': 39168}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909187000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'system_cpu_usage': {'one_min_avg': 1.42, 'five_min_avg': 2.02, 'fifteen_min_avg': 2.11, 'cpu_share': 6.25, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175632, 'used': 10317464, 'free': 363332, 'buffers': 94208, 'cached': 5400628, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'ContrailConfig': {'__T': 1754909358195550, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.56"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-1."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-1."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 16855184944691956189, "uuid_lslong": 12031691376148785163}, "enable": true, "created": "2025-08-11T10:47:02.142790", "last_modified": "2025-08-11T10:47:02.142790", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"e9e9a567-93e0-49dd-a6f9-278f5a7aa80b"'}}} __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('contrail-api is not connected to config db\n{\'Nod...uid\': \'"e9e9a567-93e0-49dd-a6f9-278f5a7aa80b"\'}}}',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/scripts/analytics/test_analytics.py", line 335, in test_verify_process_status_config self.analytics_obj.verify_process_and_connection_infos_config() File "/contrail-test/tcutils/collector/analytics_tests.py", line 4178, in verify_process_and_connection_infos_config assert result, 'contrail-api is not connected to config db\n' + str(ops_inspect) AssertionError: contrail-api is not connected to config db {'NodeStatus': {'process_status': [{'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up'}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-device-manager', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer Connection State updated'}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754916280064757, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 24502700, 'partition_space_available_1k': 56587784, 'percentage_partition_space_used': 30}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 139856, 'cpu_share': 0.0, 'mem_res': 113596}, 'contrail-schema': {'mem_virt': 89560, 'cpu_share': 0.0, 'mem_res': 80084}, 'contrail-svc-monitor': {'mem_virt': 88628, 'cpu_share': 0.0, 'mem_res': 79076}, 'contrail-device-manager': {'mem_virt': 191180, 'cpu_share': 0.0, 'mem_res': 177128}, 'contrail-config-nodemgr': {'mem_virt': 45788, 'cpu_share': 0.01, 'mem_res': 39168}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909187000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'system_cpu_usage': {'one_min_avg': 1.42, 'five_min_avg': 2.02, 'fifteen_min_avg': 2.11, 'cpu_share': 6.25, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175632, 'used': 10317464, 'free': 363332, 'buffers': 94208, 'cached': 5400628, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'ContrailConfig': {'__T': 1754909358195550, 'deleted': False, 'elements': {'config_node_ip_address': '"10.0.0.56"', 'display_name': '"cn-jenkins-deploy-platform-ansible-os-3803-1."', 'fq_name': '["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3803-1."]', 'id_perms': '{"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 16855184944691956189, "uuid_lslong": 12031691376148785163}, "enable": true, "created": "2025-08-11T10:47:02.142790", "last_modified": "2025-08-11T10:47:02.142790", "description": null, "user_visible": true, "creator": null}', 'parent_type': '"global-system-config"', 'parent_uuid': '"14566f66-4a5d-4eb7-a5fb-d1e94ba3ff8a"', 'perms2': '{"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}', 'uuid': '"e9e9a567-93e0-49dd-a6f9-278f5a7aa80b"'}}} 2025-08-11 12:45:22,653 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:45:22,660 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:45:22,667 - DEBUG - No XMPP flaps were noticed during the test 2025-08-11 12:45:22,667 - INFO - 2025-08-11 12:45:22,667 - INFO - END TEST : test_verify_process_status_config : FAILED[0:00:02] 2025-08-11 12:45:22,667 - INFO - -------------------------------------------------------------------------------- 2025-08-11 12:58:45,261 - DEBUG - [10.0.0.56]: Running cmd : hostname 2025-08-11 12:58:45,443 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1 2025-08-11 12:58:45,444 - DEBUG - [10.0.0.56]: Running cmd : hostname -f 2025-08-11 12:58:45,503 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 12:58:45,503 - DEBUG - [10.0.0.56]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:58:45,596 - DEBUG - Output : NAMES contrail_test_izHBaL8iO analytics_snmp_nodemgr_1 analytics_snmp_topology_1 analytics_snmp_snmp-collector_1 analytics_alarm_alarm-gen_1 analytics_alarm_nodemgr_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_collector_1 analytics_api_1 analytics_database_query-engine_1 analytics_database_nodemgr_1 analytics_database_cassandra_1 control_named_1 control_control_1 control_nodemgr_1 control_dns_1 webui_job_1 webui_web_1 config_devicemgr_1 config_api_1 config_schema_1 config_dnsmasq_1 config_nodemgr_1 config_svcmonitor_1 config_database_rabbitmq_1 config_database_nodemgr_1 config_database_zookeeper_1 config_database_cassandra_1 redis_redis_1 barbican_worker barbican_keystone_listener barbican_api horizon heat_engine heat_api_cfn heat_api neutron_server nova_novncproxy nova_conductor nova_api nova_scheduler glance_api keystone keystone_fernet keystone_ssh rabbitmq memcached mariadb cron kolla_toolbox fluentd 2025-08-11 12:58:45,597 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.0.0.56 | head -n 1 | awk '{print $2}' 2025-08-11 12:58:45,663 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 12:58:45,663 - DEBUG - [10.0.0.56]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-11 12:58:45,722 - DEBUG - Output : 127.0.0.1/8 10.0.0.56/24 10.20.0.17/24 172.17.0.1/16 2025-08-11 12:58:45,723 - DEBUG - [10.0.0.56]: Running cmd : getent hosts 10.20.0.17 | head -n 1 | awk '{print $2}' 2025-08-11 12:58:45,782 - DEBUG - Output : 2025-08-11 12:58:45,783 - DEBUG - [10.0.0.38]: Running cmd : hostname 2025-08-11 12:58:45,954 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 12:58:45,955 - DEBUG - [10.0.0.38]: Running cmd : hostname -f 2025-08-11 12:58:46,014 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 12:58:46,015 - DEBUG - [10.0.0.38]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:58:46,103 - DEBUG - Output : NAMES analytics_snmp_snmp-collector_1 analytics_snmp_nodemgr_1 analytics_snmp_topology_1 analytics_alarm_alarm-gen_1 analytics_alarm_nodemgr_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_api_1 analytics_collector_1 analytics_database_nodemgr_1 analytics_database_query-engine_1 analytics_database_cassandra_1 control_named_1 control_dns_1 control_nodemgr_1 control_control_1 webui_web_1 webui_job_1 config_devicemgr_1 config_api_1 config_dnsmasq_1 config_schema_1 config_svcmonitor_1 config_nodemgr_1 config_database_nodemgr_1 config_database_rabbitmq_1 config_database_cassandra_1 config_database_zookeeper_1 redis_redis_1 2025-08-11 12:58:46,104 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-11 12:58:46,162 - DEBUG - Output : 127.0.0.1/8 10.0.0.38/24 10.20.0.254/24 172.17.0.1/16 2025-08-11 12:58:46,163 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.0.0.38 | head -n 1 | awk '{print $2}' 2025-08-11 12:58:46,236 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 12:58:46,236 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.20.0.254 | head -n 1 | awk '{print $2}' 2025-08-11 12:58:46,299 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 12:58:46,300 - DEBUG - [10.0.0.22]: Running cmd : hostname 2025-08-11 12:58:46,470 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3 2025-08-11 12:58:46,471 - DEBUG - [10.0.0.22]: Running cmd : hostname -f 2025-08-11 12:58:46,528 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3. 2025-08-11 12:58:46,529 - DEBUG - [10.0.0.22]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:58:46,608 - DEBUG - Output : NAMES analytics_snmp_snmp-collector_1 analytics_snmp_topology_1 analytics_snmp_nodemgr_1 analytics_alarm_nodemgr_1 analytics_alarm_alarm-gen_1 analytics_alarm_kafka_1 analytics_nodemgr_1 analytics_api_1 analytics_collector_1 analytics_database_query-engine_1 analytics_database_nodemgr_1 analytics_database_cassandra_1 control_named_1 control_nodemgr_1 control_dns_1 control_control_1 webui_web_1 webui_job_1 config_devicemgr_1 config_nodemgr_1 config_svcmonitor_1 config_api_1 config_schema_1 config_dnsmasq_1 config_database_rabbitmq_1 config_database_nodemgr_1 config_database_zookeeper_1 config_database_cassandra_1 redis_redis_1 2025-08-11 12:58:46,608 - DEBUG - [10.0.0.22]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-08-11 12:58:46,682 - DEBUG - Output : 127.0.0.1/8 10.0.0.22/24 10.20.0.25/24 172.17.0.1/16 2025-08-11 12:58:46,682 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.0.0.22 | head -n 1 | awk '{print $2}' 2025-08-11 12:58:46,740 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3. 2025-08-11 12:58:46,740 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.20.0.25 | head -n 1 | awk '{print $2}' 2025-08-11 12:58:46,800 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-os-3803-3 2025-08-11 12:58:46,801 - DEBUG - [10.0.0.19]: Running cmd : hostname 2025-08-11 12:58:46,975 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1 2025-08-11 12:58:46,975 - DEBUG - [10.0.0.19]: Running cmd : hostname -f 2025-08-11 12:58:47,033 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 12:58:47,033 - DEBUG - [10.0.0.19]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:58:47,133 - DEBUG - Output : NAMES vrouter_vrouter-agent_1 vrouter_nodemgr_1 rsyslogd_rsyslogd_1 nova_compute nova_libvirt nova_ssh cron kolla_toolbox fluentd 2025-08-11 12:58:47,134 - DEBUG - [10.0.0.19]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-08-11 12:58:47,193 - DEBUG - Output : 10.20.0.193/24 2025-08-11 12:58:47,193 - DEBUG - [10.0.0.19]: Running cmd : getent hosts 10.20.0.193 | head -n 1 | awk '{print $2}' 2025-08-11 12:58:47,267 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-1 2025-08-11 12:58:47,268 - DEBUG - [10.0.0.41]: Running cmd : hostname 2025-08-11 12:58:47,451 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 12:58:47,451 - DEBUG - [10.0.0.41]: Running cmd : hostname -f 2025-08-11 12:58:47,515 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 12:58:47,515 - DEBUG - [10.0.0.41]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-08-11 12:58:47,614 - DEBUG - Output : NAMES vrouter_vrouter-agent_1 vrouter_nodemgr_1 rsyslogd_rsyslogd_1 nova_compute nova_libvirt nova_ssh cron kolla_toolbox fluentd 2025-08-11 12:58:47,614 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-08-11 12:58:47,683 - DEBUG - Output : 10.20.0.129/24 2025-08-11 12:58:47,683 - DEBUG - [10.0.0.41]: Running cmd : getent hosts 10.20.0.129 | head -n 1 | awk '{print $2}' 2025-08-11 12:58:47,758 - DEBUG - Output : an-jenkins-deploy-platform-ansible-os-3803-2 2025-08-11 12:58:48,567 - DEBUG - Not creating keypair since it exists 2025-08-11 12:58:48,941 - INFO - Using existing project ['default-domain', 'admin'](74e57a64-053a-4ab9-9a41-98124cb3c978) 2025-08-11 12:58:49,108 - INFO - ================================================================================ 2025-08-11 12:58:49,108 - INFO - STARTING TEST : test_verify_process_status_config 2025-08-11 12:58:49,109 - INFO - TEST DESCRIPTION : Test to validate process_status-Config 2025-08-11 12:58:49,725 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:58:49,732 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:58:49,738 - DEBUG - Nothing to compare xmpp stats {'10.0.0.19': {'10.20.0.25': '1', '10.20.0.254': '1'}, '10.0.0.41': {'10.20.0.17': '1', '10.20.0.25': '1'}} with 2025-08-11 12:58:49,738 - INFO - Initial checks done. Running the testcase now 2025-08-11 12:58:49,738 - INFO - 2025-08-11 12:58:49,738 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 12:58:50,025 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 12:58:50,087 - ERROR - cn-jenkins-deploy-platform-ansible-os-3803-1. 2025-08-11 12:58:50,088 - ERROR - {'NodeStatus': {'process_status': [{'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up'}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-device-manager', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer Connection State updated'}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754917120156265, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 24625404, 'partition_space_available_1k': 56465080, 'percentage_partition_space_used': 30}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 106312, 'cpu_share': 0.02, 'mem_res': 93828}, 'contrail-schema': {'mem_virt': 89552, 'cpu_share': 0.01, 'mem_res': 80084}, 'contrail-svc-monitor': {'mem_virt': 88596, 'cpu_share': 0.0, 'mem_res': 79076}, 'contrail-device-manager': {'mem_virt': 191280, 'cpu_share': 0.0, 'mem_res': 177152}, 'contrail-config-nodemgr': {'mem_virt': 45812, 'cpu_share': 0.01, 'mem_res': 39168}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909187000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'system_cpu_usage': {'one_min_avg': 2.0, 'five_min_avg': 2.08, 'fifteen_min_avg': 2.08, 'cpu_share': 9.5, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175632, 'used': 10342064, 'free': 463708, 'buffers': 94904, 'cached': 5274956, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'UVEAlarms': {'__T': 1754916470751207, 'alarms': [{'type': 'default-global-system-config:system-defined-conf-incorrect', 'alarm_rules': {'or_list': [{'and_list': [{'condition': {'operation': '==', 'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'variables': []}, 'match': [{'json_operand1_value': 'null', 'json_variables': {}}]}]}]}, 'ack': False, 'timestamp': 1754916470751037, 'severity': 1, 'token': 'eyJob3N0X2lwIjogIjEwLjAuMC4zOCIsICJodHRwX3BvcnQiOiA1OTk1LCAidGltZXN0YW1wIjogMTc1NDkxNjQ3MDc1MTAzN30=', 'description': 'ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.'}]}} 2025-08-11 12:58:50,088 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat 2025-08-11 12:58:50,144 - ERROR - cn-jenkins-deploy-platform-ansible-os-3803-2. 2025-08-11 12:58:50,144 - ERROR - {'NodeStatus': {'process_status': [{'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754917122313146, 'system_mem_usage': {'total': 16175640, 'used': 6932652, 'free': 246908, 'buffers': 287148, 'cached': 8708932, 'node_type': 'config-node'}, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9318784, 'partition_space_available_1k': 71771700, 'percentage_partition_space_used': 11}}, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909189000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 99844, 'cpu_share': 0.02, 'mem_res': 96880}, 'contrail-schema': {'mem_virt': 77220, 'cpu_share': 0.0, 'mem_res': 74252}, 'contrail-svc-monitor': {'mem_virt': 75744, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 102808, 'cpu_share': 0.0, 'mem_res': 84872}, 'contrail-config-nodemgr': {'mem_virt': 52168, 'cpu_share': 0.01, 'mem_res': 40192}}, 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}', 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'system_cpu_usage': {'one_min_avg': 2.84, 'five_min_avg': 1.75, 'fifteen_min_avg': 1.47, 'cpu_share': 7.5, 'node_type': 'config-node'}, 'deleted': False}, 'UVEAlarms': {'__T': 1754916470740432, 'alarms': [{'type': 'default-global-system-config:system-defined-conf-incorrect', 'alarm_rules': {'or_list': [{'and_list': [{'condition': {'operation': '==', 'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'variables': []}, 'match': [{'json_operand1_value': 'null', 'json_variables': {}}]}]}]}, 'ack': False, 'timestamp': 1754916470740255, 'severity': 1, 'token': 'eyJob3N0X2lwIjogIjEwLjAuMC4zOCIsICJodHRwX3BvcnQiOiA1OTk1LCAidGltZXN0YW1wIjogMTc1NDkxNjQ3MDc0MDI1NX0=', 'description': 'ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.'}]}} 2025-08-11 12:58:50,144 - WARNING - No object found for module contrail-device-manager 2025-08-11 12:58:50,144 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat 2025-08-11 12:58:50,202 - ERROR - cn-jenkins-deploy-platform-ansible-os-3803-3. 2025-08-11 12:58:50,203 - ERROR - {'NodeStatus': {'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 9341772, 'partition_space_available_1k': 71748712, 'percentage_partition_space_used': 12}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 102624, 'cpu_share': 0.02, 'mem_res': 98312}, 'contrail-schema': {'mem_virt': 77756, 'cpu_share': 0.0, 'mem_res': 74824}, 'contrail-svc-monitor': {'mem_virt': 75776, 'cpu_share': 0.0, 'mem_res': 72692}, 'contrail-device-manager': {'mem_virt': 102756, 'cpu_share': 0.0, 'mem_res': 84820}, 'contrail-config-nodemgr': {'mem_virt': 52248, 'cpu_share': 0.01, 'mem_res': 40196}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, '__T': 1754917120370930, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'process_status': [{'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}], 'description': None}], 'system_cpu_usage': {'one_min_avg': 1.08, 'five_min_avg': 0.97, 'fifteen_min_avg': 0.99, 'cpu_share': 6.5, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175640, 'used': 7012288, 'free': 211008, 'buffers': 285904, 'cached': 8666440, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'UVEAlarms': {'__T': 1754916470734301, 'alarms': [{'type': 'default-global-system-config:system-defined-conf-incorrect', 'alarm_rules': {'or_list': [{'and_list': [{'condition': {'operation': '==', 'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'variables': []}, 'match': [{'json_operand1_value': 'null', 'json_variables': {}}]}]}]}, 'ack': False, 'timestamp': 1754916470734106, 'severity': 1, 'token': 'eyJob3N0X2lwIjogIjEwLjAuMC4zOCIsICJodHRwX3BvcnQiOiA1OTk1LCAidGltZXN0YW1wIjogMTc1NDkxNjQ3MDczNDEwNn0=', 'description': 'ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.'}]}} 2025-08-11 12:58:50,203 - WARNING - No object found for module contrail-device-manager 2025-08-11 12:58:50,203 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 12:58:50,228 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.56:8086 servers NOT UP 2025-08-11 12:58:50,228 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.38:8086 servers NOT UP 2025-08-11 12:58:50,228 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.22:8086 servers UP 2025-08-11 12:58:50,229 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.56:2181 servers UP 2025-08-11 12:58:50,229 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.56:9041 servers UP 2025-08-11 12:58:50,229 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-api module connection to 10.0.0.56:5673 servers UP 2025-08-11 12:58:50,229 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-2.?flat 2025-08-11 12:58:50,250 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-api module connection to 10.0.0.56:8086 servers UP 2025-08-11 12:58:50,250 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-api module connection to 10.0.0.56:2181 servers UP 2025-08-11 12:58:50,251 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-api module connection to 10.0.0.56:9041 servers UP 2025-08-11 12:58:50,251 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-2.:contrail-api module connection to 10.0.0.56:5673 servers UP 2025-08-11 12:58:50,251 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-3.?flat 2025-08-11 12:58:50,274 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-api module connection to 10.0.0.56:8086 servers UP 2025-08-11 12:58:50,274 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-api module connection to 10.0.0.56:2181 servers UP 2025-08-11 12:58:50,275 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-api module connection to 10.0.0.56:9041 servers UP 2025-08-11 12:58:50,275 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-3.:contrail-api module connection to 10.0.0.56:5673 servers UP 2025-08-11 12:58:50,275 - DEBUG - Requesting: http://10.0.0.56:8081/analytics/uves/config-node/cn-jenkins-deploy-platform-ansible-os-3803-1.?flat 2025-08-11 12:58:50,299 - WARNING - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-device-manager module connection to 10.0.0.56:8086 servers NOT UP 2025-08-11 12:58:50,300 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-device-manager module connection to 10.0.0.38:8086 servers UP 2025-08-11 12:58:50,300 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-device-manager module connection to 10.0.0.56:2181 servers UP 2025-08-11 12:58:50,300 - INFO - cn-jenkins-deploy-platform-ansible-os-3803-1.:contrail-device-manager module connection to 10.0.0.56:9041 servers UP 2025-08-11 12:58:50,941 - ERROR - AssertionError Python 3.6.8: /usr/bin/python3 Mon Aug 11 12:58:50 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/scripts/analytics/test_analytics.py in test_verify_process_status_config(self=) 333 334 ''' 335 self.analytics_obj.verify_process_and_connection_infos_config() 336 337 @test.attr(type=['cb_sanity', 'sanity', 'dev_sanity_dpdk']) self = self.analytics_obj = self.analytics_obj.verify_process_and_connection_infos_config = > /contrail-test/tcutils/collector/analytics_tests.py in verify_process_and_connection_infos_config(self=) 4235 result_cassandra = result_cassandra or self.verify_connection_infos( 4236 ops_inspect, 'contrail-device-manager', server, node=cfgm) 4237 assert result, 'contrail-device-manager is not connected to config db\n' + str(ops_inspect) 4238 4239 result = False result = False builtinstr = ops_inspect = {'NodeStatus': {'process_status': [{'module_id':...fmap as ContrailConfig is missing/incorrect.'}]}} AssertionError: contrail-device-manager is not connected to config db {'NodeStatus': {'process_status': [{'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up'}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-device-manager', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer Connection State updated'}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754917120156265, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 24625404, 'partition_space_available_1k': 56465080, 'percentage_partition_space_used': 30}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 106312, 'cpu_share': 0.02, 'mem_res': 93828}, 'contrail-schema': {'mem_virt': 89552, 'cpu_share': 0.01, 'mem_res': 80084}, 'contrail-svc-monitor': {'mem_virt': 88596, 'cpu_share': 0.0, 'mem_res': 79076}, 'contrail-device-manager': {'mem_virt': 191280, 'cpu_share': 0.0, 'mem_res': 177152}, 'contrail-config-nodemgr': {'mem_virt': 45812, 'cpu_share': 0.01, 'mem_res': 39168}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909187000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'system_cpu_usage': {'one_min_avg': 2.0, 'five_min_avg': 2.08, 'fifteen_min_avg': 2.08, 'cpu_share': 9.5, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175632, 'used': 10342064, 'free': 463708, 'buffers': 94904, 'cached': 5274956, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'UVEAlarms': {'__T': 1754916470751207, 'alarms': [{'type': 'default-global-system-config:system-defined-conf-incorrect', 'alarm_rules': {'or_list': [{'and_list': [{'condition': {'operation': '==', 'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'variables': []}, 'match': [{'json_operand1_value': 'null', 'json_variables': {}}]}]}]}, 'ack': False, 'timestamp': 1754916470751037, 'severity': 1, 'token': 'eyJob3N0X2lwIjogIjEwLjAuMC4zOCIsICJodHRwX3BvcnQiOiA1OTk1LCAidGltZXN0YW1wIjogMTc1NDkxNjQ3MDc1MTAzN30=', 'description': 'ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.'}]}} __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ("contrail-device-manager is not connected to conf...fmap as ContrailConfig is missing/incorrect.'}]}}",) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/scripts/analytics/test_analytics.py", line 335, in test_verify_process_status_config self.analytics_obj.verify_process_and_connection_infos_config() File "/contrail-test/tcutils/collector/analytics_tests.py", line 4237, in verify_process_and_connection_infos_config assert result, 'contrail-device-manager is not connected to config db\n' + str(ops_inspect) AssertionError: contrail-device-manager is not connected to config db {'NodeStatus': {'process_status': [{'module_id': 'contrail-svc-monitor', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up'}], 'description': None}, {'module_id': 'contrail-config-nodemgr', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.56:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}], 'description': None}, {'module_id': 'contrail-device-manager', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer Connection State updated'}], 'description': None}, {'module_id': 'contrail-schema', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.38:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'ApiServer', 'name': 'ApiServer', 'server_addrs': ['10.0.0.56,10.0.0.38,10.0.0.22:8082'], 'status': 'Up', 'description': 'ApiServer'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}], 'description': None}, {'module_id': 'contrail-api', 'instance_id': '0', 'state': 'Functional', 'connection_infos': [{'type': 'Zookeeper', 'name': 'Zookeeper', 'server_addrs': ['10.0.0.56:2181', '10.0.0.38:2181', '10.0.0.22:2181'], 'status': 'Up', 'description': None}, {'type': 'Collector', 'name': 'Collector', 'server_addrs': ['10.0.0.22:8086'], 'status': 'Up', 'description': 'ClientInit to Established on EvSandeshCtrlMessageRecv'}, {'type': 'Database', 'name': 'Cassandra', 'server_addrs': ['10.0.0.56:9041', '10.0.0.38:9041', '10.0.0.22:9041'], 'status': 'Up', 'description': None}, {'type': 'Database', 'name': 'RabbitMQ', 'server_addrs': ['10.0.0.56:5673', '10.0.0.38:5673', '10.0.0.22:5673'], 'status': 'Up', 'description': None}, {'type': 'Generic Connection', 'name': 'Keystone', 'server_addrs': ['http://10.0.0.56:5000/v3'], 'status': 'Up', 'description': None}], 'description': None}], '__T': 1754917120156265, 'disk_usage_info': {'/dev/vda1': {'partition_type': 'ext4', 'partition_space_used_1k': 24625404, 'partition_space_available_1k': 56465080, 'percentage_partition_space_used': 30}}, 'process_mem_cpu_usage': {'contrail-api': {'mem_virt': 106312, 'cpu_share': 0.02, 'mem_res': 93828}, 'contrail-schema': {'mem_virt': 89552, 'cpu_share': 0.01, 'mem_res': 80084}, 'contrail-svc-monitor': {'mem_virt': 88596, 'cpu_share': 0.0, 'mem_res': 79076}, 'contrail-device-manager': {'mem_virt': 191280, 'cpu_share': 0.0, 'mem_res': 177152}, 'contrail-config-nodemgr': {'mem_virt': 45812, 'cpu_share': 0.01, 'mem_res': 39168}}, 'installed_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'running_package_version': 'master.1.9.8.6.1-master_1_9_8_6_1.el7', 'deleted': False, 'process_info': [{'process_name': 'contrail-api', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-schema', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-svc-monitor', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909187000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-device-manager', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}, {'process_name': 'contrail-config-nodemgr', 'process_state': 'PROCESS_STATE_RUNNING', 'start_count': 1, 'stop_count': 0, 'exit_count': 0, 'last_start_time': '1754909188000000', 'last_stop_time': None, 'last_exit_time': None, 'core_file_list': []}], 'system_cpu_info': {'num_socket': 1, 'num_cpu': 4, 'num_core_per_socket': 4, 'num_thread_per_core': 1}, 'system_cpu_usage': {'one_min_avg': 2.0, 'five_min_avg': 2.08, 'fifteen_min_avg': 2.08, 'cpu_share': 9.5, 'node_type': 'config-node'}, 'system_mem_usage': {'total': 16175632, 'used': 10342064, 'free': 463708, 'buffers': 94904, 'cached': 5274956, 'node_type': 'config-node'}, 'build_info': '{"build-info" : [{"build-version" : "master.1.9.8.6.1", "build-time" : "2025-08-08 08:34:31.991562", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10162-1.novalocal", "build-id" : "master.1.9.8.6.1", "build-number" : "master_1_9_8_6_1.el7"}]}'}, 'UVEAlarms': {'__T': 1754916470751207, 'alarms': [{'type': 'default-global-system-config:system-defined-conf-incorrect', 'alarm_rules': {'or_list': [{'and_list': [{'condition': {'operation': '==', 'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'variables': []}, 'match': [{'json_operand1_value': 'null', 'json_variables': {}}]}]}]}, 'ack': False, 'timestamp': 1754916470751037, 'severity': 1, 'token': 'eyJob3N0X2lwIjogIjEwLjAuMC4zOCIsICJodHRwX3BvcnQiOiA1OTk1LCAidGltZXN0YW1wIjogMTc1NDkxNjQ3MDc1MTAzN30=', 'description': 'ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.'}]}} 2025-08-11 12:58:50,941 - DEBUG - Requesting: http://10.0.0.19:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:58:50,948 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-08-11 12:58:50,955 - DEBUG - No XMPP flaps were noticed during the test 2025-08-11 12:58:50,955 - INFO - 2025-08-11 12:58:50,955 - INFO - END TEST : test_verify_process_status_config : FAILED[0:00:01] 2025-08-11 12:58:50,955 - INFO - --------------------------------------------------------------------------------