06/04/2026 05:58:37.427 7fc6d3dcc500 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 06/04/2026 05:58:37.427 7fc6d3dcc500 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 06/04/2026 05:58:37.427 7fc6d3dcc500 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 06/04/2026 05:58:37.438 7fc6d0c1dea0 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 06/04/2026 05:58:37.439 7fc6d0c1dea0 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 06/04/2026 05:58:37.440 7fc6d0eb62c0 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 06/04/2026 05:58:37.441 7fc6d0eb62c0 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 05:58:37.442 7fc6d0eb62c0 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/04/2026 05:58:37.442 7fc6d0eb62c0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 06/04/2026 05:58:37.442 7fc6d0eb62c0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:37.442 7fc6d0eb62c0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/04/2026 05:58:37.442 7fc6d0eb62c0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:37.442 7fc6d0eb62c0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:37.443 7fc6d0eb62c0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1780552717441636 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 06/04/2026 05:58:37.443 7fc6d0eb62c0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:37.531 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 102029652879406316243814808530695156654460077445821595481757711424595762803426, 'start': '1780552715000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 79534218748291287350524527422647441384607880338749258876606898632219481546115, 'start': '1780552715000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 74033779540427618952818493285246900269783349817922965944369316895960214067819, 'start': '1780552715000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 06/04/2026 05:58:37.576 7fc6d0eb6360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:37.577 7fc6d0eb6360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. build_info = {"build-info": [{"build-version": "v0+nightly.r25.1", "build-time": "2026-05-28 00:47:15.690553", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15063-1.novalocal", "build-id": "v0+nightly.r25.1", "build-number": "202605280007"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> >> 06/04/2026 05:58:37.577 7fc6d0eb6360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:37.577 7fc6d3dcc500 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780552717577815, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 05:58:37.578 7fc6d3dcc500 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780552717578201, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780552715000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780552715000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780552715000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info": [{"build-version": "v0+nightly.r25.1", "build-time": "2026-05-28 00:47:15.690553", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15063-1.novalocal", "build-id": "v0+nightly.r25.1", "build-number": "202605280007"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 05:58:37.580 7fc6d0eb6360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:37.580 7fc6d0eb6360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/04/2026 05:58:37.580 7fc6d0eb6360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:37.580 7fc6d0eb6360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:37.580 7fc6d0eb6360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1780552715000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1780552715000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1780552715000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info": [{"build-version": "v0+nightly.r25.1", "build-time": "2026-05-28 00:47:15.690553", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15063-1.novalocal", "build-id": "v0+nightly.r25.1", "build-number": "202605280007"}]} >> 06/04/2026 05:58:37.580 7fc6d0eb6360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:37.827 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 05:58:38.006 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 06/04/2026 05:58:38.008 7fc6d0c669a0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:38.008 7fc6d0c669a0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. database_usage = [ << disk_space_used_1k = 7483528 disk_space_available_1k = 73606956 analytics_db_size_1k = 108 >>, ] >> 06/04/2026 05:58:38.008 7fc6d0c669a0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:38.390 7fc6d3dcc500 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1 06/04/2026 05:58:38.390 7fc6d3dcc500 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780552718390573, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 05:58:38.391 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:38.392 7fc6d0c66e00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/04/2026 05:58:38.392 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:41.442 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 05:58:41.443 7fc6d0c66e00 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 05:58:41.443 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 05:58:41.443 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:58:41.443 7fc6d0c66e00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 05:58:41.443 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:58:41.444 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 05:58:41.444 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 05:58:41.444 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:41.444 7fc6d0c66e00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 05:58:41.444 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:41.444 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:41.444 7fc6d0c66e00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1780552717441636 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/04/2026 05:58:41.445 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:41.479 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780552721479625, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 05:58:41.479 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 06/04/2026 05:58:41.481 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:41.481 7fc6d0c66e00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] CassandraStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. cassandra_compaction_task = << pending_compaction_tasks = 0 >> thread_pool_stats = [ << pool_name = ReadStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = CompactionExecutor active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MutationStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MemtableFlushWriter active = 0 pending = 0 all_time_blocked = 0 >>, ] >> 06/04/2026 05:58:41.481 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:42.474 7fc6d3dcc500 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 06/04/2026 05:58:45.445 7fc6d0c66cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 05:58:45.447 7fc6d0c66cc0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 05:58:45.447 7fc6d0c66cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 05:58:45.448 7fc6d0c66cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:58:45.448 7fc6d0c66cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 05:58:45.448 7fc6d0c66cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:58:45.448 7fc6d0c66cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 05:58:45.448 7fc6d0c66cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 05:58:45.449 7fc6d0c66cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:45.449 7fc6d0c66cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 05:58:45.449 7fc6d0c66cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:45.449 7fc6d0c66cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:45.449 7fc6d0c66cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1780552717441636 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/04/2026 05:58:45.449 7fc6d0c66cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:48.080 7fc6d3dcc500 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780552728080176, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780552715000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780552715000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780552715000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info": [{"build-version": "v0+nightly.r25.1", "build-time": "2026-05-28 00:47:15.690553", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15063-1.novalocal", "build-id": "v0+nightly.r25.1", "build-number": "202605280007"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 05:58:48.080 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7538812, partition_space_available_1k=73551672, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1282620, cpu_share=0.0, mem_res=1274060, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=7332, cpu_share=0.0, mem_res=5896, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=50996, cpu_share=0.0, mem_res=41336, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=4610044, free=4450388, buffers=191280, cached=7225812, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.86, five_min_avg=0.97, fifteen_min_avg=0.41, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 05:58:48.080 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:48.080 7fc6d0c66e00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1780552715000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1780552715000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1780552715000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info": [{"build-version": "v0+nightly.r25.1", "build-time": "2026-05-28 00:47:15.690553", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15063-1.novalocal", "build-id": "v0+nightly.r25.1", "build-number": "202605280007"}]} >> 06/04/2026 05:58:48.081 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:48.081 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:48.081 7fc6d0c66e00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 7538812 partition_space_available_1k = 73551672 percentage_partition_space_used = 9 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1282620 cpu_share = 0.0 mem_res = 1274060 >>contrail-query-engine : << mem_virt = 7332 cpu_share = 0.0 mem_res = 5896 >>contrail-database-nodemgr : << mem_virt = 50996 cpu_share = 0.0 mem_res = 41336 >> } system_mem_usage = << total = 16175640 used = 4610044 free = 4450388 buffers = 191280 cached = 7225812 node_type = database-node >> system_cpu_usage = << one_min_avg = 1.86 five_min_avg = 0.97 fifteen_min_avg = 0.41 cpu_share = 0 node_type = database-node >> >> 06/04/2026 05:58:48.081 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:49.449 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 05:58:49.450 7fc6d0c66e00 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 05:58:49.451 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 05:58:49.451 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:58:49.451 7fc6d0c66e00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 05:58:49.451 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:58:49.451 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 05:58:49.451 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 05:58:49.451 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:49.452 7fc6d0c66e00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 05:58:49.452 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:49.452 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:49.452 7fc6d0c66e00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1780552717441636 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/04/2026 05:58:49.452 7fc6d0c66e00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:53.452 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 05:58:53.453 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 05:58:53.453 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 05:58:53.453 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:58:53.453 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 05:58:53.454 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:58:53.454 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 05:58:53.454 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 05:58:53.454 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:53.454 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 05:58:53.454 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:53.454 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:53.455 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1780552717441636 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/04/2026 05:58:53.455 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:57.458 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 05:58:57.459 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 05:58:57.459 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 05:58:57.459 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:58:57.459 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 05:58:57.460 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:58:57.460 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 05:58:57.460 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 05:58:57.460 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:57.460 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 05:58:57.460 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:57.460 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:58:57.460 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1780552717441636 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/04/2026 05:58:57.460 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:59:01.462 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 05:59:01.463 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 05:59:01.463 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 05:59:01.463 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:59:01.463 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 05:59:01.463 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:59:01.463 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 05:59:01.464 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 05:59:01.464 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:59:01.464 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 05:59:01.464 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:59:01.464 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:59:01.464 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1780552717441636 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/04/2026 05:59:01.464 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:59:05.466 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 05:59:05.467 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 05:59:05.467 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 05:59:05.467 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:59:05.467 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 05:59:05.467 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:59:05.467 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 05:59:05.468 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 05:59:05.468 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:59:05.468 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 05:59:05.468 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:59:05.468 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:59:05.468 7fc6d17cde00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1780552717441636 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/04/2026 05:59:05.468 7fc6d17cde00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:59:09.468 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 05:59:09.470 7fc6d0eb65e0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 05:59:09.470 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 05:59:09.470 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:59:09.470 7fc6d0eb65e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 05:59:09.470 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:59:09.470 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 05:59:09.471 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 05:59:09.471 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:59:09.471 7fc6d0eb65e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 05:59:09.471 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:59:09.471 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:59:09.471 7fc6d0eb65e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1780552717441636 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/04/2026 05:59:09.471 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 05:59:13.475 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 05:59:13.476 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 06/04/2026 05:59:13.476 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 05:59:13.476 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:59:13.477 7fc6d0eb65e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 05:59:13.477 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 05:59:13.477 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 06/04/2026 05:59:13.477 7fc6d0eb65e0 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 06/04/2026 05:59:13.478 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 06/04/2026 05:59:13.478 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 05:59:13.479 7fc6d0eb65e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 05:59:27.381 7fc6d17cd900 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001016 06/04/2026 05:59:27.381 7fc6d17cd900 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 06/04/2026 05:59:27.382 7fc6d19b19a0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 06/04/2026 05:59:27.382 7fc6d19b19a0 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 06/04/2026 05:59:27.382 7fc6d19b19a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 06/04/2026 05:59:27.382 7fc6d19b19a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 06/04/2026 05:59:27.382 7fc6d19b19a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 05:59:27.382 7fc6d19b19a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/04/2026 05:59:27.383 7fc6d19b19a0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 06/04/2026 05:59:27.383 7fc6d0eb6f40 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 06/04/2026 05:59:27.384 7fc6d0eb6f40 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 06/04/2026 05:59:37.377 7fc6d17cd900 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001582 06/04/2026 05:59:37.378 7fc6d17cd900 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 06/04/2026 05:59:37.379 7fc6d19b19a0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 06/04/2026 05:59:37.379 7fc6d19b19a0 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 4 06/04/2026 05:59:37.380 7fc6d19b19a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 06/04/2026 05:59:37.380 7fc6d19b19a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 06/04/2026 05:59:37.380 7fc6d19b19a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 05:59:37.381 7fc6d19b19a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/04/2026 05:59:37.381 7fc6d0eb6680 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 06/04/2026 05:59:37.382 7fc6d0eb6680 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 06/04/2026 05:59:37.790 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 05:59:40.101 7fc6d3dcc500 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780552780101383, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.33:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 05:59:43.202 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780552783202230, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 05:59:49.086 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8710432, partition_space_available_1k=72380052, percentage_partition_space_used=11, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1378224, cpu_share=0.39, mem_res=1360464, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11776, cpu_share=0.0, mem_res=9820, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51000, cpu_share=0.01, mem_res=41332, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5771488, free=2337612, buffers=227668, cached=8141140, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.63, five_min_avg=1.26, fifteen_min_avg=0.54, cpu_share=9.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:00:37.807 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:00:45.758 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8737504, partition_space_available_1k=72352980, percentage_partition_space_used=11, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1391828, cpu_share=0.21, mem_res=1371116, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12120, cpu_share=0.0, mem_res=10168, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51000, cpu_share=0.01, mem_res=41332, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6195088, free=1885756, buffers=229632, cached=8167800, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.72, five_min_avg=1.24, fifteen_min_avg=0.58, cpu_share=7.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:01:37.800 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:01:45.702 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8740700, partition_space_available_1k=72349784, percentage_partition_space_used=11, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1366632, cpu_share=0.11, mem_res=1344844, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12304, cpu_share=0.0, mem_res=10352, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51008, cpu_share=0.01, mem_res=41332, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6231232, free=1846052, buffers=229676, cached=8171372, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.35, five_min_avg=1.21, fifteen_min_avg=0.61, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:02:37.809 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:02:45.880 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8743668, partition_space_available_1k=72346816, percentage_partition_space_used=11, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1396336, cpu_share=0.13, mem_res=1373068, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12476, cpu_share=0.0, mem_res=10524, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51008, cpu_share=0.01, mem_res=41332, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6287132, free=1783768, buffers=229752, cached=8175688, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.03, five_min_avg=1.13, fifteen_min_avg=0.62, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:02:59.394 7fc6d17cd180 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 06:03:15.193 7fc6d0eb6680 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 06:03:37.800 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:03:45.753 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8886880, partition_space_available_1k=72203604, percentage_partition_space_used=11, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1384896, cpu_share=0.15, mem_res=1360408, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12620, cpu_share=0.0, mem_res=10668, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51028, cpu_share=0.01, mem_res=41332, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6295108, free=1622684, buffers=232804, cached=8325644, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.12, five_min_avg=1.15, fifteen_min_avg=0.66, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:04:37.832 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:04:45.580 7fc6d3dcc500 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8890476, partition_space_available_1k=72200008, percentage_partition_space_used=11, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1391100, cpu_share=0.12, mem_res=1364620, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12656, cpu_share=0.0, mem_res=10704, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51032, cpu_share=0.01, mem_res=41332, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6298956, free=1615188, buffers=232856, cached=8329236, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.93, five_min_avg=1.09, fifteen_min_avg=0.67, cpu_share=5.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:05:35.978 7f53f14d1580 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 06/04/2026 06:05:35.978 7f53f14d1580 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 06/04/2026 06:05:35.978 7f53f14d1580 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 06/04/2026 06:05:36.010 7f53ee6ee180 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 06/04/2026 06:05:36.010 7f53ee6ee180 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 06/04/2026 06:05:36.014 7f53ee6ee540 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 06/04/2026 06:05:36.016 7f53ee6ee540 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 06:05:36.017 7f53ee6ee540 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/04/2026 06:05:36.017 7f53ee6ee540 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 06/04/2026 06:05:36.017 7f53ee6ee540 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:36.018 7f53ee6ee540 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/04/2026 06:05:36.018 7f53ee6ee540 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:36.018 7f53ee6ee540 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:36.018 7f53ee6ee540 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 06/04/2026 06:05:36.018 7f53ee6ee540 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:39.452 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 102029652879406316243814808530695156654460077445821595481757711424595762803426, 'start': '1780553134000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 79534218748291287350524527422647441384607880338749258876606898632219481546115, 'start': '1780553133000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 74033779540427618952818493285246900269783349817922965944369316895960214067819, 'start': '1780553133000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 06/04/2026 06:05:39.716 7f53ee6eeae0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:39.727 7f53ee6eeae0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. build_info = {"build-info": [{"build-version": "v0+nightly.r25.1", "build-time": "2026-05-28 00:47:15.690553", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15063-1.novalocal", "build-id": "v0+nightly.r25.1", "build-number": "202605280007"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> >> 06/04/2026 06:05:39.728 7f53ee6eeae0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:39.728 7f53f14d1580 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780553139728487, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:05:39.733 7f53f14d1580 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780553139733294, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780553134000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780553133000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780553133000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info": [{"build-version": "v0+nightly.r25.1", "build-time": "2026-05-28 00:47:15.690553", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15063-1.novalocal", "build-id": "v0+nightly.r25.1", "build-number": "202605280007"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:05:39.746 7f53ee6eeae0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:39.746 7f53ee6eeae0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/04/2026 06:05:39.746 7f53ee6eeae0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:39.746 7f53ee6eeae0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:39.758 7f53ee6eeae0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1780553134000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1780553133000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1780553133000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info": [{"build-version": "v0+nightly.r25.1", "build-time": "2026-05-28 00:47:15.690553", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15063-1.novalocal", "build-id": "v0+nightly.r25.1", "build-number": "202605280007"}]} >> 06/04/2026 06:05:39.758 7f53ee6eeae0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:40.022 7f53ee5de680 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:05:40.051 7f53ee5de680 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 06/04/2026 06:05:40.056 7f53ee5de680 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:05:40.056 7f53ee5de680 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:05:40.056 7f53ee5de680 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:05:40.056 7f53ee5de680 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:05:40.056 7f53ee5de680 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 06/04/2026 06:05:40.057 7f53ee5de680 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 06/04/2026 06:05:40.067 7f53ee5de680 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 06/04/2026 06:05:40.067 7f53ee5de680 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 06:05:40.067 7f53ee5de680 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 06:05:41.615 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:05:44.417 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 06/04/2026 06:05:44.419 7f53ee6eee00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 06:05:46.922 7f53ee5c89a0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 06/04/2026 06:05:46.922 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 06/04/2026 06:05:46.923 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 06/04/2026 06:05:46.923 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:46.923 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 06/04/2026 06:05:46.923 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:46.923 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:46.923 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/04/2026 06:05:46.923 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:48.415 7f53f14d1580 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1 06/04/2026 06:05:48.415 7f53f14d1580 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780553148415919, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.33:8086'], status='Down', description='ClientInit to Idle on EvTcpClose')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:05:48.424 7f53ee6eea40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:48.424 7f53ee6eea40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 06/04/2026 06:05:48.424 7f53ee6eea40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:05:50.926 7f53ee5c8680 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:05:50.927 7f53ee5c8680 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 06/04/2026 06:05:50.929 7f53ee5c8680 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:05:50.929 7f53ee5c8680 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:05:50.929 7f53ee5c8680 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:05:50.931 7f53ee5c8680 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:05:50.931 7f53ee5c8680 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 06/04/2026 06:05:50.931 7f53ee5c8680 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 06/04/2026 06:05:50.932 7f53ee5c8680 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 06/04/2026 06:05:50.932 7f53ee5c8680 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 06:05:50.932 7f53ee5c8680 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 06:05:50.946 7f53ee6ee9a0 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001017 06/04/2026 06:05:50.947 7f53ee6ee9a0 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 06/04/2026 06:05:50.947 7f53ee6eed60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 06/04/2026 06:05:50.947 7f53ee6eed60 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 06/04/2026 06:05:50.948 7f53ee6eed60 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 06/04/2026 06:05:50.948 7f53ee6eed60 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 06:05:50.948 7f53ee6eed60 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/04/2026 06:05:50.948 7f53ee6eed60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 06/04/2026 06:05:50.949 7f53ee5c85e0 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 06/04/2026 06:05:50.949 7f53ee5c85e0 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 06/04/2026 06:05:56.366 7f53f14d1580 [contrail-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 2 06/04/2026 06:06:06.846 7f53f14d1580 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: timed out 06/04/2026 06:06:11.918 7f53f14d1580 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780553171918673, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780553134000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780553133000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780553133000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info": [{"build-version": "v0+nightly.r25.1", "build-time": "2026-05-28 00:47:15.690553", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15063-1.novalocal", "build-id": "v0+nightly.r25.1", "build-number": "202605280007"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:06:11.919 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=9270108, partition_space_available_1k=71820376, percentage_partition_space_used=11, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1359408, cpu_share=0.0, mem_res=1351832, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=9464, cpu_share=0.0, mem_res=7700, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39656, cpu_share=0.0, mem_res=36972, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6077672, free=1416648, buffers=242516, cached=8816228, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=14.45, five_min_avg=4.77, fifteen_min_avg=1.96, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:06:40.157 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:06:41.103 7f53f14d1580 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780553201103145, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.38:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:06:45.932 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780553205932487, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:06:45.932 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 06/04/2026 06:06:51.628 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10360588, partition_space_available_1k=70729896, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1390696, cpu_share=0.56, mem_res=1373736, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10176, cpu_share=0.0, mem_res=8400, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39840, cpu_share=0.02, mem_res=37136, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6088704, free=168440, buffers=306524, cached=9951420, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=11.38, five_min_avg=5.15, fifteen_min_avg=2.2, cpu_share=19.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:07:39.934 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:07:44.222 7f53ee6ee9a0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 06/04/2026 06:07:44.222 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[Established] 06/04/2026 06:07:44.224 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 06/04/2026 06:07:44.225 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[Established] -> State[Connect] 06/04/2026 06:07:44.225 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:07:44.225 7f53ee5de860 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Established to Connect on EvTcpClose >>, ] description = Collector:Collector[Established to Connect on EvTcpClose] connection down >>, ] >> 06/04/2026 06:07:44.225 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:07:44.225 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 06/04/2026 06:07:44.225 7f53ee5de860 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 06/04/2026 06:07:44.226 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 06/04/2026 06:07:44.226 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 06:07:44.226 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 06:07:44.263 7f53ee5de400 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 06/04/2026 06:07:44.263 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 06/04/2026 06:07:44.263 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 06/04/2026 06:07:44.263 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:07:44.264 7f53ee5de860 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 06/04/2026 06:07:44.264 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:07:44.264 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:07:44.264 7f53ee5de860 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:07:44.264 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:07:48.265 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:07:48.266 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 06/04/2026 06:07:48.267 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:07:48.267 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:07:48.267 7f53ee5de860 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:07:48.267 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:07:48.267 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 06/04/2026 06:07:48.267 7f53ee5de860 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 06/04/2026 06:07:48.268 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 06/04/2026 06:07:48.268 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 06:07:48.268 7f53ee5de860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 06:07:48.587 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11119884, partition_space_available_1k=69970600, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1371132, cpu_share=0.16, mem_res=1352708, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10756, cpu_share=0.0, mem_res=8956, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39984, cpu_share=0.01, mem_res=37136, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=4871536, free=1507784, buffers=310632, cached=9864576, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.87, five_min_avg=4.66, fifteen_min_avg=2.21, cpu_share=11.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:07:48.588 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 06:07:56.077 7f53ee5de400 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 06/04/2026 06:07:56.078 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 06/04/2026 06:07:56.079 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 06/04/2026 06:07:56.079 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:07:56.079 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 06/04/2026 06:07:56.079 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:07:56.079 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:07:56.079 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:07:56.079 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:00.082 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:00.083 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:00.083 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:00.084 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:00.084 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:00.084 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:00.084 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:00.084 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:00.084 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:00.084 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:00.084 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:00.084 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:00.085 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:00.085 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:04.088 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:04.090 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:04.090 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:04.090 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:04.090 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:04.090 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:04.091 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:04.091 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:04.091 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:04.091 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:04.091 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:04.092 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:04.092 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:04.092 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:08.094 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:08.095 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:08.095 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:08.095 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:08.095 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:08.095 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:08.095 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:08.096 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:08.096 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:08.096 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:08.096 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:08.096 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:08.096 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:08.096 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:12.096 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:12.098 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:12.098 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:12.099 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:12.099 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:12.099 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:12.099 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:12.099 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:12.100 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:12.100 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:12.100 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:12.100 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:12.100 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:12.100 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:16.103 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:16.104 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:16.105 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:16.105 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:16.105 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:16.105 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:16.105 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:16.106 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:16.106 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:16.106 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:16.106 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:16.106 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:16.106 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:16.106 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:20.106 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:20.107 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:20.107 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:20.107 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:20.107 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:20.107 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:20.107 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:20.108 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:20.108 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:20.108 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:20.108 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:20.108 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:20.108 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:20.108 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:24.109 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:24.111 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:24.111 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:24.111 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:24.111 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:24.111 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:24.111 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:24.112 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:24.112 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:24.112 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:24.112 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:24.112 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:24.112 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:24.113 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:28.112 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:28.114 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:28.114 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:28.114 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:28.114 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:28.114 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:28.114 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:28.115 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:28.115 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:28.115 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:28.115 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:28.115 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:28.115 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:28.115 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:32.118 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:32.119 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:32.119 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:32.119 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:32.119 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:32.119 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:32.120 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:32.120 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:32.120 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:32.120 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:32.120 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:32.121 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:32.121 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:32.121 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:36.122 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:36.123 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:36.123 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:36.123 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:36.124 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:36.124 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:36.124 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:36.124 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:36.124 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:36.124 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:36.124 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:36.124 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:36.125 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:36.125 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:39.922 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:08:40.068 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:40.068 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. database_usage = [ << disk_space_used_1k = 10925572 disk_space_available_1k = 70164912 analytics_db_size_1k = 8920 >>, ] >> 06/04/2026 06:08:40.068 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:40.124 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:40.125 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:40.125 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:40.125 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:40.125 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:40.126 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:40.126 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:40.126 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:40.126 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:40.126 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:40.126 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:40.126 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:40.127 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:40.127 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:44.126 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:44.127 7f53ee5de400 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:44.127 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:44.128 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:44.128 7f53ee5de400 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:44.129 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:44.129 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:44.129 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:44.130 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:44.130 7f53ee5de400 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:44.130 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:44.131 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:44.131 7f53ee5de400 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:44.131 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:48.129 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:48.130 7f53ee5de400 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:48.131 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:48.131 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:48.131 7f53ee5de400 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:48.131 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:48.131 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:48.132 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:48.132 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:48.132 7f53ee5de400 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:48.132 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:48.132 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:48.133 7f53ee5de400 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:48.133 7f53ee5de400 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:49.010 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10930396, partition_space_available_1k=70160088, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1344852, cpu_share=0.08, mem_res=1326648, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11408, cpu_share=0.0, mem_res=9416, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40048, cpu_share=0.01, mem_res=37136, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=4303548, free=2220352, buffers=317716, cached=9712616, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.09, five_min_avg=4.07, fifteen_min_avg=2.16, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:08:49.010 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:49.010 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 10930396 partition_space_available_1k = 70160088 percentage_partition_space_used = 13 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1344852 cpu_share = 0.08 mem_res = 1326648 >>contrail-query-engine : << mem_virt = 11408 cpu_share = 0.0 mem_res = 9416 >>contrail-database-nodemgr : << mem_virt = 40048 cpu_share = 0.01 mem_res = 37136 >> } system_mem_usage = << total = 16175640 used = 4303548 free = 2220352 buffers = 317716 cached = 9712616 node_type = database-node >> system_cpu_usage = << one_min_avg = 3.09 five_min_avg = 4.07 fifteen_min_avg = 2.16 cpu_share = 4.75 node_type = database-node >> >> 06/04/2026 06:08:49.011 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:52.132 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:52.134 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:52.134 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:52.135 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:52.136 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:52.136 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:52.136 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:52.137 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:52.137 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:52.138 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:52.138 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:52.138 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:52.139 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:52.139 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:56.139 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:08:56.140 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:08:56.140 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:08:56.140 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:56.140 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:08:56.141 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:08:56.141 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:08:56.141 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:08:56.141 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:56.141 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:08:56.141 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:56.141 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:08:56.141 7f53eefd1a40 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:08:56.142 7f53eefd1a40 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:00.144 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:09:00.145 7f53ee5c8c20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:09:00.146 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:09:00.146 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:00.146 7f53ee5c8c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:09:00.146 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:00.146 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:09:00.146 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:09:00.147 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:00.147 7f53ee5c8c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:09:00.147 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:00.147 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:00.147 7f53ee5c8c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:09:00.147 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:04.150 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:09:04.151 7f53ee5c8c20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:09:04.152 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:09:04.152 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:04.152 7f53ee5c8c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:09:04.152 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:04.152 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:09:04.152 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:09:04.153 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:04.153 7f53ee5c8c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:09:04.153 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:04.153 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:04.153 7f53ee5c8c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:09:04.153 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:08.154 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:09:08.155 7f53ee5c8c20 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:09:08.156 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:09:08.156 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:08.156 7f53ee5c8c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:09:08.156 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:08.156 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:09:08.157 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:09:08.157 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:08.157 7f53ee5c8c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:09:08.158 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:08.158 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:08.158 7f53ee5c8c20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:09:08.158 7f53ee5c8c20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:12.156 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:09:12.157 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:09:12.157 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:09:12.157 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:12.158 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:09:12.158 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:12.158 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:09:12.158 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:09:12.158 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:12.158 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:09:12.158 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:12.158 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:12.158 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:09:12.158 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:16.162 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:09:16.163 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:09:16.163 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:09:16.163 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:16.164 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:09:16.164 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:16.164 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:09:16.164 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:09:16.164 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:16.164 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:09:16.165 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:16.165 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:16.165 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:09:16.165 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:20.166 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:09:20.167 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:09:20.168 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:09:20.168 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:20.168 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:09:20.168 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:20.168 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:09:20.169 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:09:20.169 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:20.169 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:09:20.169 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:20.169 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:20.169 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:09:20.169 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:24.170 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:09:24.172 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/04/2026 06:09:24.172 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:09:24.172 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:24.172 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:09:24.172 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:24.172 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/04/2026 06:09:24.173 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/04/2026 06:09:24.173 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:24.173 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/04/2026 06:09:24.173 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:24.173 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:24.173 7f53ee5c8180 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1780553136017078 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 06/04/2026 06:09:24.173 7f53ee5c8180 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:09:28.174 7f53ee62f040 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:09:28.175 7f53ee62f040 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 06/04/2026 06:09:28.175 7f53ee62f040 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:09:28.175 7f53ee62f040 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:28.175 7f53ee62f040 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:09:28.175 7f53ee62f040 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:09:28.175 7f53ee62f040 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 06/04/2026 06:09:28.176 7f53ee62f040 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 06/04/2026 06:09:28.176 7f53ee62f040 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 06/04/2026 06:09:28.176 7f53ee62f040 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 06:09:28.176 7f53ee62f040 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 06:09:28.182 7f53ee5c8e00 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001017 06/04/2026 06:09:28.182 7f53ee5c8e00 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 06/04/2026 06:09:28.182 7f53ee62f040 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 06/04/2026 06:09:28.183 7f53ee62f040 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 06/04/2026 06:09:28.183 7f53ee62f040 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 06/04/2026 06:09:28.183 7f53ee62f040 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 06/04/2026 06:09:28.183 7f53ee62f040 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 06:09:28.183 7f53ee62f040 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/04/2026 06:09:28.183 7f53ee62f040 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 06/04/2026 06:09:28.183 7f53ee62f9a0 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 06/04/2026 06:09:28.185 7f53ee62f9a0 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 06/04/2026 06:09:40.032 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:09:46.469 7f53ee5c8e00 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001582 06/04/2026 06:09:46.470 7f53ee5c8e00 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 06/04/2026 06:09:46.471 7f53ee62f040 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 06/04/2026 06:09:46.471 7f53ee62f040 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 4 06/04/2026 06:09:46.471 7f53ee62f040 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 06/04/2026 06:09:46.471 7f53ee62f040 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 06/04/2026 06:09:46.472 7f53ee62f040 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 06:09:46.472 7f53ee62f040 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/04/2026 06:09:46.472 7f53ee5c82c0 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 06/04/2026 06:09:46.473 7f53ee5c82c0 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 06/04/2026 06:09:48.997 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11153604, partition_space_available_1k=69936880, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1368860, cpu_share=0.14, mem_res=1349092, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11684, cpu_share=0.0, mem_res=9524, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40148, cpu_share=0.01, mem_res=37176, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6238112, free=253208, buffers=326740, cached=9694956, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.02, five_min_avg=3.84, fifteen_min_avg=2.19, cpu_share=12.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:10:39.977 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:10:47.880 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11166748, partition_space_available_1k=69923736, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1395788, cpu_share=0.19, mem_res=1373640, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11844, cpu_share=0.0, mem_res=9636, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40616, cpu_share=0.01, mem_res=37184, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6251240, free=277960, buffers=323492, cached=9629888, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.06, five_min_avg=3.37, fifteen_min_avg=2.13, cpu_share=6.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:11:39.970 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:11:48.014 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10770816, partition_space_available_1k=70319668, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1378492, cpu_share=0.15, mem_res=1354920, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11368, cpu_share=0.0, mem_res=9168, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40620, cpu_share=0.01, mem_res=37184, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5923392, free=1023728, buffers=326136, cached=9210144, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.94, five_min_avg=3.06, fifteen_min_avg=2.1, cpu_share=8.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:12:39.983 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:12:47.825 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10778736, partition_space_available_1k=70311748, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1383604, cpu_share=0.17, mem_res=1356700, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12328, cpu_share=0.0, mem_res=10132, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40624, cpu_share=0.01, mem_res=37188, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6154320, free=783716, buffers=326164, cached=9217888, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.89, five_min_avg=2.79, fifteen_min_avg=2.07, cpu_share=6.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:13:39.998 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:13:47.684 7f53f14d1580 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10787088, partition_space_available_1k=70303396, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1398636, cpu_share=0.15, mem_res=1368660, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12740, cpu_share=0.0, mem_res=10544, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40636, cpu_share=0.01, mem_res=37196, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6261868, free=642100, buffers=326516, cached=9251892, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.81, five_min_avg=2.58, fifteen_min_avg=2.03, cpu_share=7.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:14:28.918 7f53ee6ee5e0 [contrail-database-nodemgr] [DEBUG]: DBG: event_process_state:contrail-query-engine,group:contrail-query-engine,state:PROCESS_STATE_EXITED 06/04/2026 06:14:28.919 7f53ee6ee5e0 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780553668919329, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780553134000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_EXITED', start_count=1, stop_count=0, exit_count=1, last_start_time='1780553133000000', last_stop_time='', last_exit_time='1780553668919265', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780553133000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info": [{"build-version": "v0+nightly.r25.1", "build-time": "2026-05-28 00:47:15.690553", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15063-1.novalocal", "build-id": "v0+nightly.r25.1", "build-number": "202605280007"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:14:53.127 7fdfe4c12a40 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 06/04/2026 06:14:53.127 7fdfe4c12a40 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 06/04/2026 06:14:53.127 7fdfe4c12a40 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 06/04/2026 06:14:53.137 7fdfe1948ea0 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 06/04/2026 06:14:53.137 7fdfe1948ea0 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 06/04/2026 06:14:53.141 7fdfe18802c0 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 06/04/2026 06:14:53.142 7fdfe18802c0 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 06:14:53.145 7fdfe18802c0 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/04/2026 06:14:53.145 7fdfe18802c0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 06/04/2026 06:14:53.145 7fdfe18802c0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:14:53.146 7fdfe18802c0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/04/2026 06:14:53.146 7fdfe18802c0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:14:53.146 7fdfe18802c0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:14:53.146 7fdfe18802c0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1780553693142801 collector_name = collector_ip = collector_list = [ 10.0.0.33:8086, 10.0.0.50:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 06/04/2026 06:14:53.147 7fdfe18802c0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:14:53.268 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 38216199311192776388336222233154135486425175173821781499225447283538407396655, 'start': '1780553690000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 7225963810554498839115198822103704414453824594801597857089528987361123802328, 'start': '1780553691000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 92024120111869224381508699249992128330750384123692826902429801059134933150560, 'start': '1780553691000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 06/04/2026 06:14:53.321 7fdfe1880360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:14:53.322 7fdfe1880360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2026-06-04 01:03:50.985224", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15148-1.novalocal", "build-id": "v0+nightly", "build-number": "202606040011"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> >> 06/04/2026 06:14:53.322 7fdfe1880360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:14:53.323 7fdfe4c12a40 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780553693322893, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:14:53.323 7fdfe4c12a40 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780553693323350, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780553690000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780553691000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780553691000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info": [{"build-version": "v0+nightly", "build-time": "2026-06-04 01:03:50.985224", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15148-1.novalocal", "build-id": "v0+nightly", "build-number": "202606040011"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:14:53.338 7fdfe1880360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:14:53.338 7fdfe1880360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/04/2026 06:14:53.338 7fdfe1880360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:14:53.339 7fdfe1880360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:14:53.339 7fdfe1880360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1780553690000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1780553691000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1780553691000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2026-06-04 01:03:50.985224", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15148-1.novalocal", "build-id": "v0+nightly", "build-number": "202606040011"}]} >> 06/04/2026 06:14:53.339 7fdfe1880360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:14:53.635 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:14:53.936 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 06/04/2026 06:14:53.937 7fdfe1880ea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:14:53.938 7fdfe1880ea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. database_usage = [ << disk_space_used_1k = 11028844 disk_space_available_1k = 70061640 analytics_db_size_1k = 17504 >>, ] >> 06/04/2026 06:14:53.938 7fdfe1880ea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:14:54.438 7fdfe4c12a40 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1 06/04/2026 06:14:54.438 7fdfe4c12a40 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780553694438408, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:14:54.439 7fdfe1880ea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:14:54.440 7fdfe1880ea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/04/2026 06:14:54.440 7fdfe1880ea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/04/2026 06:14:56.292 7fdfe4c12a40 [contrail-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 2 06/04/2026 06:14:57.143 7fdfe1880ea0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/04/2026 06:14:57.144 7fdfe1880ea0 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 06/04/2026 06:14:57.145 7fdfe1880ea0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/04/2026 06:14:57.145 7fdfe1880ea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:14:57.145 7fdfe1880ea0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-5888-3. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/04/2026 06:14:57.145 7fdfe1880ea0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/04/2026 06:14:57.145 7fdfe1880ea0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 06/04/2026 06:14:57.145 7fdfe1880ea0 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 06/04/2026 06:14:57.145 7fdfe1880ea0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 06/04/2026 06:14:57.146 7fdfe1880ea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 06:14:57.146 7fdfe1880ea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/04/2026 06:14:57.152 7fdfe16abf40 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001017 06/04/2026 06:14:57.152 7fdfe16abf40 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 06/04/2026 06:14:57.152 7fdfe16abd60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 06/04/2026 06:14:57.154 7fdfe16abd60 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 06/04/2026 06:14:57.154 7fdfe16abd60 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 06/04/2026 06:14:57.154 7fdfe16abd60 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 06:14:57.154 7fdfe16abd60 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/04/2026 06:14:57.155 7fdfe16abd60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 06/04/2026 06:14:57.155 7fdfe16ab040 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 06/04/2026 06:14:57.157 7fdfe16ab040 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 06/04/2026 06:15:00.332 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: [2026-06-04 06:14:59,980] Replication factor is 1. No repair is needed for keyspace 'ContrailAnalyticsCql' 06/04/2026 06:15:05.387 7fdfe4c12a40 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780553705387311, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780553690000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780553691000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1780553691000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info": [{"build-version": "v0+nightly", "build-time": "2026-06-04 01:03:50.985224", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-15148-1.novalocal", "build-id": "v0+nightly", "build-number": "202606040011"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:15:05.387 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11030424, partition_space_available_1k=70060060, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1273004, cpu_share=0.0, mem_res=1262152, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=9564, cpu_share=0.0, mem_res=7660, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55384, cpu_share=0.0, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6071704, free=559900, buffers=336520, cached=9511068, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.29, five_min_avg=2.84, fifteen_min_avg=2.17, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:15:53.531 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:15:54.426 7fdfe4c12a40 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780553754426162, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.33:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:15:56.920 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780553756920083, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:15:56.920 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 06/04/2026 06:16:01.964 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11033060, partition_space_available_1k=70057424, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1377276, cpu_share=0.35, mem_res=1348292, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10316, cpu_share=0.0, mem_res=8412, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55644, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6387000, free=273212, buffers=335056, cached=9484128, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.0, five_min_avg=2.58, fifteen_min_avg=2.12, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:16:53.524 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:17:01.502 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11036420, partition_space_available_1k=70054064, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1370968, cpu_share=0.14, mem_res=1340992, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10496, cpu_share=0.0, mem_res=8588, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55580, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6532544, free=324132, buffers=332272, cached=9291224, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.33, five_min_avg=2.28, fifteen_min_avg=2.04, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:17:53.522 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:18:01.472 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11039296, partition_space_available_1k=70051188, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1371844, cpu_share=0.13, mem_res=1340612, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10508, cpu_share=0.0, mem_res=8600, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55588, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6566256, free=287376, buffers=332292, cached=9294244, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.25, five_min_avg=2.06, fifteen_min_avg=1.98, cpu_share=6.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:18:53.559 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:19:01.330 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11042220, partition_space_available_1k=70048264, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1392792, cpu_share=0.17, mem_res=1360364, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10592, cpu_share=0.0, mem_res=8684, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55568, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6615852, free=282512, buffers=332064, cached=9250140, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.92, five_min_avg=1.82, fifteen_min_avg=1.9, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:19:53.555 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:20:01.442 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11045576, partition_space_available_1k=70044908, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1413732, cpu_share=0.13, mem_res=1379724, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10644, cpu_share=0.0, mem_res=8732, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55584, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6635440, free=271600, buffers=332156, cached=9241444, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.89, five_min_avg=1.64, fifteen_min_avg=1.83, cpu_share=6.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:20:53.603 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:21:01.393 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11049328, partition_space_available_1k=70041156, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1386516, cpu_share=0.12, mem_res=1350812, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10600, cpu_share=0.0, mem_res=8688, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55580, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6654108, free=267080, buffers=332152, cached=9227484, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.03, five_min_avg=1.53, fifteen_min_avg=1.78, cpu_share=5.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:21:53.579 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:22:01.332 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11052468, partition_space_available_1k=70038016, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1418712, cpu_share=0.14, mem_res=1381272, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10632, cpu_share=0.0, mem_res=8720, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55572, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6664044, free=289424, buffers=331888, cached=9195772, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.39, five_min_avg=1.56, fifteen_min_avg=1.78, cpu_share=6.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:22:53.593 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:23:01.312 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11055432, partition_space_available_1k=70035052, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1397620, cpu_share=0.12, mem_res=1359116, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10648, cpu_share=0.0, mem_res=8736, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55580, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6660152, free=297928, buffers=331896, cached=9191240, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.24, five_min_avg=1.49, fifteen_min_avg=1.74, cpu_share=6.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:23:53.602 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:24:01.202 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11058344, partition_space_available_1k=70032140, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1383652, cpu_share=0.1, mem_res=1343956, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10652, cpu_share=0.0, mem_res=8740, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55584, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6656488, free=298364, buffers=331904, cached=9194416, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.28, five_min_avg=1.43, fifteen_min_avg=1.7, cpu_share=5.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:24:53.609 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:25:01.453 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11061412, partition_space_available_1k=70029072, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1392996, cpu_share=0.11, mem_res=1352228, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10704, cpu_share=0.0, mem_res=8792, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55584, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6668784, free=282620, buffers=331904, cached=9197912, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.36, five_min_avg=1.41, fifteen_min_avg=1.68, cpu_share=6.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:25:53.614 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:26:01.481 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11064872, partition_space_available_1k=70025612, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1389160, cpu_share=0.11, mem_res=1346568, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10708, cpu_share=0.0, mem_res=8796, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55588, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6659636, free=301764, buffers=331836, cached=9188072, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.93, five_min_avg=1.29, fifteen_min_avg=1.61, cpu_share=5.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:26:53.621 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:27:01.320 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11063220, partition_space_available_1k=70027264, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1407560, cpu_share=0.11, mem_res=1363368, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10712, cpu_share=0.0, mem_res=8800, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55572, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6679368, free=315992, buffers=331816, cached=9154636, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.12, five_min_avg=1.27, fifteen_min_avg=1.59, cpu_share=6.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:27:53.652 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:28:01.593 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11066332, partition_space_available_1k=70024152, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1408768, cpu_share=0.1, mem_res=1363448, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10796, cpu_share=0.0, mem_res=8884, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55572, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6707660, free=297380, buffers=332016, cached=9144876, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.86, five_min_avg=1.17, fifteen_min_avg=1.53, cpu_share=5.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:28:53.630 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:29:01.479 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11068640, partition_space_available_1k=70021844, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1410208, cpu_share=0.1, mem_res=1364232, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10800, cpu_share=0.0, mem_res=8888, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55572, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6695004, free=307496, buffers=332028, cached=9147396, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.04, five_min_avg=1.14, fifteen_min_avg=1.5, cpu_share=5.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:29:46.749 7fdfe19487c0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 06:29:53.676 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:30:01.421 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11203976, partition_space_available_1k=69886508, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1409108, cpu_share=0.1, mem_res=1362480, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10904, cpu_share=0.0, mem_res=8992, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55528, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6720632, free=312904, buffers=334684, cached=9114272, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.48, five_min_avg=1.22, fifteen_min_avg=1.5, cpu_share=6.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:30:06.599 7fdfe19487c0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 06:30:53.657 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:31:01.493 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11210480, partition_space_available_1k=69880004, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1406336, cpu_share=0.1, mem_res=1358192, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10960, cpu_share=0.0, mem_res=9048, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55536, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6712932, free=311284, buffers=335000, cached=9123348, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.27, five_min_avg=1.2, fifteen_min_avg=1.47, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:31:27.228 7fdfe1880f40 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 06:31:53.685 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:32:01.834 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11215644, partition_space_available_1k=69874840, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1439276, cpu_share=0.16, mem_res=1388148, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12660, cpu_share=0.0, mem_res=10716, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55504, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6782596, free=316112, buffers=334848, cached=9049760, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.95, five_min_avg=1.39, fifteen_min_avg=1.52, cpu_share=7.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:32:53.683 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:33:01.611 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11217868, partition_space_available_1k=69872616, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1425296, cpu_share=0.14, mem_res=1371448, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12628, cpu_share=0.0, mem_res=10696, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55504, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6823384, free=280808, buffers=334764, cached=9044480, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.49, five_min_avg=1.35, fifteen_min_avg=1.49, cpu_share=8.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:33:53.720 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:34:03.102 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11230180, partition_space_available_1k=69860304, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1435480, cpu_share=0.2, mem_res=1375576, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12684, cpu_share=0.0, mem_res=10752, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55496, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6842804, free=293656, buffers=334656, cached=9012692, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.81, five_min_avg=1.7, fifteen_min_avg=1.6, cpu_share=8.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:34:53.762 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:35:01.817 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11235936, partition_space_available_1k=69854548, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1461144, cpu_share=0.17, mem_res=1396096, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12792, cpu_share=0.0, mem_res=10860, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55484, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6862748, free=302136, buffers=334320, cached=8984068, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.94, five_min_avg=1.65, fifteen_min_avg=1.59, cpu_share=8.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:35:53.785 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:36:02.078 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11249500, partition_space_available_1k=69840984, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1460404, cpu_share=0.18, mem_res=1389484, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12916, cpu_share=0.0, mem_res=10984, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55480, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6871472, free=309940, buffers=334196, cached=8967932, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.92, five_min_avg=1.67, fifteen_min_avg=1.6, cpu_share=9.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:36:53.710 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:36:57.163 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780555017162948, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=1, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:37:02.202 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11259432, partition_space_available_1k=69831052, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1466356, cpu_share=0.16, mem_res=1390980, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12964, cpu_share=0.0, mem_res=11032, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55484, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6883004, free=287472, buffers=334208, cached=8978892, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.67, five_min_avg=2.4, fifteen_min_avg=1.85, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:37:53.733 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:37:56.912 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-5888-3.', _node_type='Database', _instance_id='0', _timestamp=1780555076911983, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 06/04/2026 06:38:01.951 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11266884, partition_space_available_1k=69823600, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1474000, cpu_share=0.13, mem_res=1394992, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12964, cpu_share=0.0, mem_res=11032, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55480, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6861088, free=330408, buffers=334144, cached=8958232, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.97, five_min_avg=2.31, fifteen_min_avg=1.85, cpu_share=6.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:38:53.724 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:39:01.923 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11271808, partition_space_available_1k=69818676, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1495252, cpu_share=0.16, mem_res=1412196, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12924, cpu_share=0.0, mem_res=10988, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55476, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6908884, free=294000, buffers=334128, cached=8947032, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.4, five_min_avg=2.53, fifteen_min_avg=1.95, cpu_share=8.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:39:53.732 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:40:01.910 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11278392, partition_space_available_1k=69812092, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1495088, cpu_share=0.13, mem_res=1409216, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13000, cpu_share=0.0, mem_res=11064, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55480, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6901388, free=295348, buffers=334072, cached=8953032, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.84, five_min_avg=2.5, fifteen_min_avg=1.98, cpu_share=7.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:40:53.782 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:41:01.777 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11287320, partition_space_available_1k=69803164, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1509636, cpu_share=0.15, mem_res=1419872, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13012, cpu_share=0.0, mem_res=11076, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55460, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6968828, free=293192, buffers=334060, cached=8888452, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.86, five_min_avg=2.59, fifteen_min_avg=2.04, cpu_share=7.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:41:53.723 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:42:01.788 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11287948, partition_space_available_1k=69802536, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1515044, cpu_share=0.12, mem_res=1422944, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13020, cpu_share=0.0, mem_res=11084, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55456, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6937200, free=319228, buffers=334228, cached=8893824, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.55, five_min_avg=2.25, fifteen_min_avg=1.96, cpu_share=7.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:42:53.730 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:43:01.918 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11293316, partition_space_available_1k=69797168, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1514184, cpu_share=0.12, mem_res=1419532, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13096, cpu_share=0.0, mem_res=11160, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55468, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6931640, free=319240, buffers=334244, cached=8899400, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.01, five_min_avg=2.21, fifteen_min_avg=1.96, cpu_share=6.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:43:41.670 7fdfe19487c0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/04/2026 06:43:53.748 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:44:01.700 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11297136, partition_space_available_1k=69793348, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1519612, cpu_share=0.14, mem_res=1421188, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13252, cpu_share=0.0, mem_res=11316, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55464, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6928760, free=320612, buffers=334236, cached=8900920, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.25, five_min_avg=2.26, fifteen_min_avg=1.99, cpu_share=7.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:44:53.775 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/04/2026 06:45:01.705 7fdfe4c12a40 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-5888-3.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11299596, partition_space_available_1k=69790888, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1525224, cpu_share=0.12, mem_res=1422728, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13204, cpu_share=0.0, mem_res=11268, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=55452, cpu_share=0.01, mem_res=45784, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6939740, free=298184, buffers=334468, cached=8912460, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.6, five_min_avg=2.07, fifteen_min_avg=1.94, cpu_share=6.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/04/2026 06:45:04.039 7fdfe19487c0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map