12/18/2025 02:03:29.725 7f9cc9694140 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 12/18/2025 02:03:29.726 7f9cc9694140 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 12/18/2025 02:03:29.726 7f9cc9694140 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 12/18/2025 02:03:29.737 7f9cc6a6c360 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 12/18/2025 02:03:29.737 7f9cc6a6c360 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 12/18/2025 02:03:29.738 7f9cc6a6c720 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 12/18/2025 02:03:29.739 7f9cc6a6c720 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/18/2025 02:03:29.739 7f9cc6a6c720 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 12/18/2025 02:03:29.739 7f9cc6a6c720 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 12/18/2025 02:03:29.739 7f9cc6a6c720 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:29.740 7f9cc6a6c720 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/18/2025 02:03:29.740 7f9cc6a6c720 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:29.740 7f9cc6a6c720 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:29.740 7f9cc6a6c720 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1766023409739566 collector_name = collector_ip = collector_list = [ 10.0.0.141:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 12/18/2025 02:03:29.740 7f9cc6a6c720 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:29.911 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 75970941311352032602701568139517105502081372909975443395529962049806015062374, 'start': '1766023407000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 90252237326516111233277813199741846693462517489770337185624027555574951046691, 'start': '1766023407000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 29294582217911533512957902767307485122287492623804495612626351385576009398473, 'start': '1766023407000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 12/18/2025 02:03:29.956 7f9cc6632860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:29.956 7f9cc6632860 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2025-12-18 00:49:09.200950", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-13034-1.novalocal", "build-id": "v0+nightly", "build-number": "202512180007"}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> >> 12/18/2025 02:03:29.956 7f9cc6632860 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:29.957 7f9cc9694140 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-4899-1.', _node_type='Database', _instance_id='0', _timestamp=1766023409957493, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/18/2025 02:03:29.957 7f9cc9694140 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-4899-1.', _node_type='Database', _instance_id='0', _timestamp=1766023409957825, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1766023407000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1766023407000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1766023407000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info": [{"build-version": "v0+nightly", "build-time": "2025-12-18 00:49:09.200950", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-13034-1.novalocal", "build-id": "v0+nightly", "build-number": "202512180007"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/18/2025 02:03:29.960 7f9cc6632860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:29.960 7f9cc6632860 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/18/2025 02:03:29.960 7f9cc6632860 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:29.960 7f9cc6632860 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:29.960 7f9cc6632860 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1766023407000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1766023407000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1766023407000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2025-12-18 00:49:09.200950", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-13034-1.novalocal", "build-id": "v0+nightly", "build-number": "202512180007"}]} >> 12/18/2025 02:03:29.960 7f9cc6632860 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:30.245 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:03:30.632 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 12/18/2025 02:03:30.634 7f9cc6a6ce00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:30.634 7f9cc6a6ce00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. database_usage = [ << disk_space_used_1k = 16421588 disk_space_available_1k = 64668896 analytics_db_size_1k = 108 >>, ] >> 12/18/2025 02:03:30.634 7f9cc6a6ce00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:31.096 7f9cc9694140 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL and socket: Result: 1 12/18/2025 02:03:31.096 7f9cc9694140 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-4899-1.', _node_type='Database', _instance_id='0', _timestamp=1766023411096832, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/18/2025 02:03:31.098 7f9cc6a6ce00 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:31.098 7f9cc6a6ce00 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/18/2025 02:03:31.098 7f9cc6a6ce00 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:33.736 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-4899-1.', _node_type='Database', _instance_id='0', _timestamp=1766023413735926, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=2), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 12/18/2025 02:03:33.736 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 12/18/2025 02:03:33.738 7f9cc66124a0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:33.739 7f9cc66124a0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] CassandraStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. cassandra_compaction_task = << pending_compaction_tasks = 2 >> thread_pool_stats = [ << pool_name = ReadStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = CompactionExecutor active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MutationStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MemtableFlushWriter active = 0 pending = 0 all_time_blocked = 0 >>, ] >> 12/18/2025 02:03:33.739 7f9cc66124a0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:33.739 7f9cc66124a0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/18/2025 02:03:33.746 7f9cc66124a0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/18/2025 02:03:33.746 7f9cc66124a0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/18/2025 02:03:33.746 7f9cc66124a0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:03:33.747 7f9cc66124a0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/18/2025 02:03:33.747 7f9cc66124a0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:03:33.751 7f9cc66124a0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/18/2025 02:03:33.754 7f9cc66124a0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/18/2025 02:03:33.754 7f9cc66124a0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:33.755 7f9cc66124a0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/18/2025 02:03:33.755 7f9cc66124a0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:33.755 7f9cc66124a0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:33.755 7f9cc66124a0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1766023409739566 collector_name = collector_ip = 10.0.0.141:8086 collector_list = [ 10.0.0.141:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/18/2025 02:03:33.755 7f9cc66124a0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:35.092 7f9cc9694140 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 12/18/2025 02:03:37.749 7f9cc6612900 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/18/2025 02:03:37.751 7f9cc6612900 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/18/2025 02:03:37.751 7f9cc6612900 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/18/2025 02:03:37.751 7f9cc6612900 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:03:37.752 7f9cc6612900 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/18/2025 02:03:37.752 7f9cc6612900 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:03:37.752 7f9cc6612900 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/18/2025 02:03:37.752 7f9cc6612900 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/18/2025 02:03:37.753 7f9cc6612900 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:37.753 7f9cc6612900 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/18/2025 02:03:37.753 7f9cc6612900 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:37.753 7f9cc6612900 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:37.754 7f9cc6612900 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1766023409739566 collector_name = collector_ip = 10.0.0.141:8086 collector_list = [ 10.0.0.141:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/18/2025 02:03:37.754 7f9cc6612900 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:40.132 7f9cc9694140 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-4899-1.', _node_type='Database', _instance_id='0', _timestamp=1766023420132135, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1766023407000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1766023407000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1766023407000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info": [{"build-version": "v0+nightly", "build-time": "2025-12-18 00:49:09.200950", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-13034-1.novalocal", "build-id": "v0+nightly", "build-number": "202512180007"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/18/2025 02:03:40.132 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=16627416, partition_space_available_1k=64463068, percentage_partition_space_used=21, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1300136, cpu_share=0.0, mem_res=1290336, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=7180, cpu_share=0.0, mem_res=5660, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47612, cpu_share=0.0, mem_res=38260, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=7834260, free=7923568, buffers=665656, cached=16709140, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.45, five_min_avg=1.94, fifteen_min_avg=1.37, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:03:40.132 7f9cc66120e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:40.133 7f9cc66120e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1766023407000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1766023407000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1766023407000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info": [{"build-version": "v0+nightly", "build-time": "2025-12-18 00:49:09.200950", "build-user": "root", "build-hostname": "jenkins-build-prepare-sandbox-13034-1.novalocal", "build-id": "v0+nightly", "build-number": "202512180007"}]} >> 12/18/2025 02:03:40.133 7f9cc66120e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:40.133 7f9cc66120e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:40.133 7f9cc66120e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 16627416 partition_space_available_1k = 64463068 percentage_partition_space_used = 21 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1300136 cpu_share = 0.0 mem_res = 1290336 >>contrail-query-engine : << mem_virt = 7180 cpu_share = 0.0 mem_res = 5660 >>contrail-database-nodemgr : << mem_virt = 47612 cpu_share = 0.0 mem_res = 38260 >> } system_mem_usage = << total = 32665120 used = 7834260 free = 7923568 buffers = 665656 cached = 16709140 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.45 five_min_avg = 1.94 fifteen_min_avg = 1.37 cpu_share = 0 node_type = database-node >> >> 12/18/2025 02:03:40.133 7f9cc66120e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:41.752 7f9cc66120e0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/18/2025 02:03:41.753 7f9cc66120e0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/18/2025 02:03:41.753 7f9cc66120e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/18/2025 02:03:41.754 7f9cc66120e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:03:41.754 7f9cc66120e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/18/2025 02:03:41.754 7f9cc66120e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:03:41.754 7f9cc66120e0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/18/2025 02:03:41.754 7f9cc66120e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/18/2025 02:03:41.754 7f9cc66120e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:41.754 7f9cc66120e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/18/2025 02:03:41.754 7f9cc66120e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:41.754 7f9cc66120e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:41.755 7f9cc66120e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1766023409739566 collector_name = collector_ip = 10.0.0.141:8086 collector_list = [ 10.0.0.141:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/18/2025 02:03:41.755 7f9cc66120e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:45.754 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/18/2025 02:03:45.756 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/18/2025 02:03:45.756 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/18/2025 02:03:45.756 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:03:45.756 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/18/2025 02:03:45.756 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:03:45.756 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/18/2025 02:03:45.757 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/18/2025 02:03:45.757 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:45.757 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/18/2025 02:03:45.757 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:45.757 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:45.757 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1766023409739566 collector_name = collector_ip = 10.0.0.141:8086 collector_list = [ 10.0.0.141:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/18/2025 02:03:45.757 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:49.757 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/18/2025 02:03:49.759 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/18/2025 02:03:49.759 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/18/2025 02:03:49.759 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:03:49.759 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/18/2025 02:03:49.759 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:03:49.759 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/18/2025 02:03:49.760 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/18/2025 02:03:49.760 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:49.760 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/18/2025 02:03:49.760 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:49.760 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:49.760 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1766023409739566 collector_name = collector_ip = 10.0.0.141:8086 collector_list = [ 10.0.0.141:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/18/2025 02:03:49.760 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:53.761 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/18/2025 02:03:53.762 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/18/2025 02:03:53.762 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/18/2025 02:03:53.762 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:03:53.762 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/18/2025 02:03:53.763 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:03:53.763 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/18/2025 02:03:53.763 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/18/2025 02:03:53.764 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:53.764 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/18/2025 02:03:53.764 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:53.764 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:53.764 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1766023409739566 collector_name = collector_ip = 10.0.0.141:8086 collector_list = [ 10.0.0.141:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/18/2025 02:03:53.764 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:57.766 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/18/2025 02:03:57.767 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/18/2025 02:03:57.767 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/18/2025 02:03:57.767 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:03:57.767 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/18/2025 02:03:57.768 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:03:57.768 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/18/2025 02:03:57.768 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/18/2025 02:03:57.768 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:57.768 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/18/2025 02:03:57.768 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:57.768 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:03:57.769 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1766023409739566 collector_name = collector_ip = 10.0.0.141:8086 collector_list = [ 10.0.0.141:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/18/2025 02:03:57.769 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:04:01.768 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/18/2025 02:04:01.770 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 12/18/2025 02:04:01.771 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/18/2025 02:04:01.771 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:04:01.771 7f9cc6612360 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/18/2025 02:04:01.772 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:04:01.772 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 12/18/2025 02:04:01.773 7f9cc6612360 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 12/18/2025 02:04:01.774 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 12/18/2025 02:04:01.774 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/18/2025 02:04:01.774 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/18/2025 02:04:13.689 7f9cc66324a0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 12/18/2025 02:04:13.689 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 12/18/2025 02:04:13.691 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 12/18/2025 02:04:13.691 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:04:13.692 7f9cc74584a0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 12/18/2025 02:04:13.692 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:04:13.692 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:04:13.692 7f9cc74584a0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1766023409739566 collector_name = collector_ip = 10.0.0.141:8086 collector_list = [ 10.0.0.141:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/18/2025 02:04:13.692 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/18/2025 02:04:17.695 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/18/2025 02:04:17.696 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 12/18/2025 02:04:17.696 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/18/2025 02:04:17.697 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:04:17.697 7f9cc74584a0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4899-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.141:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/18/2025 02:04:17.697 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/18/2025 02:04:17.697 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 12/18/2025 02:04:17.697 7f9cc74584a0 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 12/18/2025 02:04:17.698 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 12/18/2025 02:04:17.698 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/18/2025 02:04:17.698 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/18/2025 02:04:17.701 7f9cc66325e0 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001017 12/18/2025 02:04:17.729 7f9cc66325e0 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 12/18/2025 02:04:17.729 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 12/18/2025 02:04:17.729 7f9cc74584a0 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 12/18/2025 02:04:17.729 7f9cc74584a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 12/18/2025 02:04:17.730 7f9cc74584a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 12/18/2025 02:04:17.730 7f9cc74584a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/18/2025 02:04:17.730 7f9cc74584a0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 12/18/2025 02:04:17.730 7f9cc74584a0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 12/18/2025 02:04:17.730 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 12/18/2025 02:04:17.732 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 12/18/2025 02:04:24.246 7f9cc66325e0 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001582 12/18/2025 02:04:24.246 7f9cc66325e0 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 12/18/2025 02:04:24.247 7f9cc738aea0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 12/18/2025 02:04:24.247 7f9cc738aea0 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 4 12/18/2025 02:04:24.248 7f9cc738aea0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 12/18/2025 02:04:24.248 7f9cc738aea0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 12/18/2025 02:04:24.248 7f9cc738aea0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/18/2025 02:04:24.248 7f9cc738aea0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 12/18/2025 02:04:24.249 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 12/18/2025 02:04:24.250 7f9cc6612360 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 12/18/2025 02:04:30.263 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:04:31.548 7f9cc9694140 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-4899-1.', _node_type='Database', _instance_id='0', _timestamp=1766023471548838, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.141:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/18/2025 02:04:34.353 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-4899-1.', _node_type='Database', _instance_id='0', _timestamp=1766023474353494, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 12/18/2025 02:04:39.416 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=17657808, partition_space_available_1k=63432676, percentage_partition_space_used=22, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1325584, cpu_share=0.37, mem_res=1309804, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11596, cpu_share=0.0, mem_res=9184, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47940, cpu_share=0.02, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=9348720, free=5463268, buffers=688884, cached=17632664, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.39, five_min_avg=2.52, fifteen_min_avg=1.6, cpu_share=4.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:05:30.293 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:05:39.071 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19471172, partition_space_available_1k=61619312, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1352872, cpu_share=0.14, mem_res=1338772, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11532, cpu_share=0.0, mem_res=9480, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40688, cpu_share=0.02, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=9369508, free=18384760, buffers=704680, cached=4675040, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.87, five_min_avg=2.69, fifteen_min_avg=1.72, cpu_share=4.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:06:30.253 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:06:38.579 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19486220, partition_space_available_1k=61604264, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1333708, cpu_share=0.11, mem_res=1316956, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11636, cpu_share=0.0, mem_res=9584, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40640, cpu_share=0.02, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=9223308, free=18395652, buffers=707616, cached=4806976, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.62, five_min_avg=2.56, fifteen_min_avg=1.74, cpu_share=3.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:07:13.239 7f9cc6a6cea0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/18/2025 02:07:15.733 7f9cc6a6cea0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/18/2025 02:07:30.295 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:07:39.013 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19674276, partition_space_available_1k=61416208, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1412540, cpu_share=0.14, mem_res=1378668, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11952, cpu_share=0.0, mem_res=9900, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40636, cpu_share=0.02, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=9993956, free=17507444, buffers=713524, cached=4917768, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.41, five_min_avg=2.52, fifteen_min_avg=1.77, cpu_share=3.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:08:29.438 7f9cc6612ea0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/18/2025 02:08:30.394 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:08:39.415 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=20938968, partition_space_available_1k=60151516, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1371060, cpu_share=0.12, mem_res=1335276, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12048, cpu_share=0.0, mem_res=9992, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40676, cpu_share=0.03, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10279816, free=15820896, buffers=748044, cached=6284032, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.73, five_min_avg=2.59, fifteen_min_avg=1.85, cpu_share=4.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:09:30.308 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:09:39.281 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=20999496, partition_space_available_1k=60090988, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1407152, cpu_share=0.24, mem_res=1365156, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17084, cpu_share=0.0, mem_res=14940, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40676, cpu_share=0.03, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10621520, free=15387912, buffers=748804, cached=6374520, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.85, five_min_avg=3.14, fifteen_min_avg=2.08, cpu_share=5.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:10:30.464 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:10:40.780 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21027356, partition_space_available_1k=60063128, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1427808, cpu_share=0.18, mem_res=1381320, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17056, cpu_share=0.0, mem_res=14912, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40676, cpu_share=0.03, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11082460, free=14890016, buffers=748864, cached=6418020, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.06, five_min_avg=3.91, fifteen_min_avg=2.42, cpu_share=5.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:11:30.330 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:11:39.577 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21056636, partition_space_available_1k=60033848, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1417912, cpu_share=0.16, mem_res=1367368, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17124, cpu_share=0.0, mem_res=14980, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40684, cpu_share=0.03, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10772004, free=15164604, buffers=749444, cached=6446672, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.21, five_min_avg=3.76, fifteen_min_avg=2.46, cpu_share=5.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:12:30.381 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:12:39.399 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21083332, partition_space_available_1k=60007152, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1441232, cpu_share=0.16, mem_res=1386696, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17240, cpu_share=0.0, mem_res=15096, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40700, cpu_share=0.03, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10854616, free=15052576, buffers=749488, cached=6475132, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.91, five_min_avg=4.43, fifteen_min_avg=2.78, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:13:30.399 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:13:39.208 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21176600, partition_space_available_1k=59913884, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1449868, cpu_share=0.14, mem_res=1391612, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17296, cpu_share=0.0, mem_res=15152, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40688, cpu_share=0.03, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11722408, free=14132268, buffers=749640, cached=6527636, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.86, five_min_avg=4.4, fifteen_min_avg=2.88, cpu_share=5.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:14:30.439 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:14:39.271 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21174516, partition_space_available_1k=59915968, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1441152, cpu_share=0.14, mem_res=1378532, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17348, cpu_share=0.0, mem_res=15204, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40692, cpu_share=0.02, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11044164, free=14774048, buffers=749680, cached=6564012, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.16, five_min_avg=4.5, fifteen_min_avg=3.0, cpu_share=4.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:15:30.384 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:15:39.300 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21196948, partition_space_available_1k=59893536, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1447572, cpu_share=0.11, mem_res=1382488, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17424, cpu_share=0.0, mem_res=15280, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40692, cpu_share=0.03, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11414724, free=14378932, buffers=749740, cached=6588540, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.11, five_min_avg=4.56, fifteen_min_avg=3.11, cpu_share=4.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:16:30.390 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:16:39.040 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21211488, partition_space_available_1k=59878996, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1458080, cpu_share=0.12, mem_res=1389860, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17484, cpu_share=0.0, mem_res=15340, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40696, cpu_share=0.02, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=11012028, free=14755164, buffers=749788, cached=6615084, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.06, five_min_avg=4.58, fifteen_min_avg=3.2, cpu_share=4.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:17:30.362 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:17:39.070 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21345912, partition_space_available_1k=59744572, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1464064, cpu_share=0.11, mem_res=1393056, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17544, cpu_share=0.0, mem_res=15400, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40700, cpu_share=0.02, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10902052, free=14739752, buffers=749872, cached=6739068, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.72, five_min_avg=4.54, fifteen_min_avg=3.28, cpu_share=5.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:18:30.409 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:18:39.211 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21356576, partition_space_available_1k=59733908, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1469168, cpu_share=0.12, mem_res=1395404, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17596, cpu_share=0.0, mem_res=15452, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40704, cpu_share=0.03, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10920160, free=14710872, buffers=749900, cached=6749752, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.53, five_min_avg=4.23, fifteen_min_avg=3.25, cpu_share=3.38, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:19:30.345 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/18/2025 02:19:35.355 7f9cc6632680 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/18/2025 02:19:39.000 7f9cc9694140 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-4899-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=21361964, partition_space_available_1k=59728520, percentage_partition_space_used=26, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1472352, cpu_share=0.12, mem_res=1395188, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17824, cpu_share=0.0, mem_res=15672, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=40708, cpu_share=0.02, mem_res=38548, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=10973992, free=14649808, buffers=749920, cached=6757028, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.98, five_min_avg=3.94, fifteen_min_avg=3.21, cpu_share=3.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/18/2025 02:20:15.621 7f9cc6632040 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map