01/20/2025 00:51:04.029 7f438e5cedf0 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 01/20/2025 00:51:04.029 7f438e5cedf0 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 01/20/2025 00:51:04.029 7f438e5cedf0 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 01/20/2025 00:51:04.036 7f4388c3e648 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 01/20/2025 00:51:04.037 7f4388c3e648 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 01/20/2025 00:51:04.037 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 01/20/2025 00:51:04.038 7f4388c3ee48 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 00:51:04.038 7f4388c3ee48 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 01/20/2025 00:51:04.038 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 01/20/2025 00:51:04.038 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:04.038 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/20/2025 00:51:04.038 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:04.038 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:04.039 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 01/20/2025 00:51:04.039 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:04.130 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 94537270109468223081139197337835364206539177810879441038868846995004006080747, 'start': '1737334262000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 110149863120647927509892561123495881608747483564226840696695047398324932250252, 'start': '1737334262000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 27691419639454453072800012030658330408095100336004145304229453583761079062662, 'start': '1737334262000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 01/20/2025 00:51:04.271 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:04.272 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-01-16 01:14:15.868302", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7545-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = nightly.r24.1.centos-nightly_r24.1_centos.el7 installed_package_version = nightly.r24.1.centos-nightly_r24.1_centos.el7 >> 01/20/2025 00:51:04.272 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:04.273 7f438e5cedf0 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737334264272916, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 00:51:04.273 7f438e5cedf0 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737334264273241, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737334262000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737334262000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737334262000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-01-16 01:14:15.868302", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7545-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 00:51:04.275 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:04.276 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/20/2025 00:51:04.276 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:04.276 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:04.276 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737334262000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737334262000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737334262000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-01-16 01:14:15.868302", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7545-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} >> 01/20/2025 00:51:04.276 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:04.514 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 00:51:04.771 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 01/20/2025 00:51:04.773 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:04.773 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. database_usage = [ << disk_space_used_1k = 6755904 disk_space_available_1k = 74334580 analytics_db_size_1k = 4 >>, ] >> 01/20/2025 00:51:04.773 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:05.133 7f438e5cedf0 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 01/20/2025 00:51:05.133 7f438e5cedf0 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737334265133696, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 00:51:05.135 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:05.135 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/20/2025 00:51:05.135 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:07.589 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737334267589787, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 00:51:07.590 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 01/20/2025 00:51:07.591 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:07.592 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] CassandraStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. cassandra_compaction_task = << pending_compaction_tasks = 0 >> thread_pool_stats = [ << pool_name = MemtableFlushWriter active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = CompactionExecutor active = 0 pending = 0 all_time_blocked = 0 >>, ] >> 01/20/2025 00:51:07.592 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:08.038 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:51:08.040 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 00:51:08.040 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:51:08.040 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:08.040 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:51:08.040 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:08.041 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 00:51:08.041 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 00:51:08.041 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:08.041 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 00:51:08.041 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:08.041 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:08.041 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 00:51:08.041 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:08.527 7f438e5cedf0 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 01/20/2025 00:51:12.040 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:51:12.041 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 00:51:12.042 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:51:12.042 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:12.042 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:51:12.042 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:12.042 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 00:51:12.042 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 00:51:12.042 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:12.043 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 00:51:12.043 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:12.043 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:12.043 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 00:51:12.043 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:13.613 7f438e5cedf0 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737334273613276, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737334262000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737334262000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737334262000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-01-16 01:14:15.868302", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7545-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 00:51:13.613 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=6756824, partition_space_available_1k=74333660, percentage_partition_space_used=8, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1214440, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=7500, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51408, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=4387272, free=5263236, buffers=239592, cached=6285532, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.96, five_min_avg=1.02, fifteen_min_avg=0.51, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 00:51:13.613 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:13.614 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737334262000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737334262000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737334262000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-01-16 01:14:15.868302", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7545-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} >> 01/20/2025 00:51:13.614 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:13.614 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:13.614 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 6756824 partition_space_available_1k = 74333660 percentage_partition_space_used = 8 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1214440 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 7500 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 51408 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175632 used = 4387272 free = 5263236 buffers = 239592 cached = 6285532 node_type = database-node >> system_cpu_usage = << one_min_avg = 0.96 five_min_avg = 1.02 fifteen_min_avg = 0.51 cpu_share = 0 node_type = database-node >> >> 01/20/2025 00:51:13.614 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:16.044 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:51:16.046 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 00:51:16.046 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:51:16.046 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:16.046 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:51:16.046 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:16.046 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 00:51:16.047 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 00:51:16.047 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:16.047 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 00:51:16.047 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:16.048 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:16.048 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 00:51:16.048 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:20.048 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:51:20.049 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 00:51:20.049 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:51:20.049 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:20.050 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:51:20.050 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:20.050 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 00:51:20.050 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 00:51:20.050 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:20.050 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 00:51:20.050 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:20.050 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:20.050 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 00:51:20.050 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:24.053 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:51:24.054 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 00:51:24.054 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:51:24.055 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:24.055 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:51:24.055 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:24.055 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 00:51:24.055 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 00:51:24.055 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:24.055 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 00:51:24.055 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:24.056 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:24.056 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 00:51:24.056 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:28.055 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:51:28.057 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 00:51:28.057 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:51:28.057 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:28.057 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:51:28.057 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:28.057 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 00:51:28.058 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 00:51:28.058 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:28.058 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 00:51:28.058 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:28.058 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:28.058 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 00:51:28.058 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:32.057 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:51:32.059 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 00:51:32.059 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:51:32.059 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:32.059 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:51:32.059 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:32.059 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 00:51:32.059 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 00:51:32.060 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:32.060 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 00:51:32.060 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:32.060 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:32.060 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 00:51:32.060 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:36.059 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:51:36.060 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 00:51:36.060 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:51:36.061 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:36.061 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:51:36.061 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:36.061 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 00:51:36.061 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 00:51:36.061 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:36.061 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 00:51:36.061 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:36.061 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:36.062 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 00:51:36.062 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:40.063 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:51:40.065 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 00:51:40.065 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:51:40.065 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:40.065 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:51:40.065 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:40.065 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 00:51:40.066 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 00:51:40.066 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:40.066 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 00:51:40.066 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:40.066 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:40.066 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 00:51:40.066 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:44.068 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:51:44.069 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 00:51:44.069 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:51:44.069 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:44.070 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:51:44.070 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:44.070 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 00:51:44.070 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 00:51:44.070 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:44.070 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 00:51:44.070 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:44.071 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:44.071 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 00:51:44.071 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:48.071 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:51:48.073 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 00:51:48.073 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:51:48.073 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:48.073 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:51:48.073 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:48.073 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 00:51:48.073 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 00:51:48.073 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:48.074 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 00:51:48.074 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:48.074 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:48.074 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 00:51:48.074 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:52.075 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:51:52.076 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 00:51:52.076 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:51:52.076 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:52.077 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:51:52.077 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:52.077 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 00:51:52.077 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 00:51:52.077 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:52.077 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 00:51:52.077 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:52.077 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:52.078 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 00:51:52.078 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:56.077 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:51:56.079 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 00:51:56.079 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:51:56.079 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:56.079 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:51:56.080 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:51:56.080 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 00:51:56.080 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 00:51:56.080 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:56.080 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 00:51:56.080 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:56.081 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:51:56.081 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 00:51:56.081 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:52:00.080 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:52:00.081 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 00:52:00.081 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:52:00.082 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:52:00.082 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:52:00.082 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:52:00.082 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 00:52:00.083 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 00:52:00.083 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:52:00.083 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 00:52:00.083 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:52:00.083 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:52:00.083 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737334264038446 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 00:52:00.083 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 00:52:04.083 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 00:52:04.085 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 01/20/2025 00:52:04.085 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 00:52:04.085 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:52:04.085 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 00:52:04.085 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 00:52:04.086 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 01/20/2025 00:52:04.086 7f4388c3ee48 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 01/20/2025 00:52:04.086 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 01/20/2025 00:52:04.086 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 00:52:04.086 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 00:52:04.511 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 00:52:04.808 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 00:52:05.277 7f438e5cedf0 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737334325277858, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.49:8086'], status='Initializing', description='Connect to ClientInit on EvTcpConnected')]), flag_infos=None, description='Collector:Collector[Connect to ClientInit on EvTcpConnected] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 00:52:05.279 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 00:52:07.386 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737334327386885, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 00:52:07.389 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 00:52:12.622 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=6771560, partition_space_available_1k=74318924, percentage_partition_space_used=8, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1300600, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=9252, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51872, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=4661684, free=4576804, buffers=246808, cached=6690336, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=0.78, five_min_avg=0.95, fifteen_min_avg=0.52, cpu_share=4.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 00:52:12.623 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 00:52:17.618 7f4389248348 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001016 01/20/2025 00:52:17.618 7f4389248348 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 01/20/2025 00:52:17.619 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 01/20/2025 00:52:17.619 7f4388c3ee48 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 01/20/2025 00:52:17.619 7f4388c3ee48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 01/20/2025 00:52:17.619 7f4388c3ee48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 01/20/2025 00:52:17.623 7f4388c3ee48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 00:52:17.623 7f4388c3ee48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 01/20/2025 00:52:17.624 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 01/20/2025 00:52:17.624 7f4387ed8048 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 01/20/2025 00:52:17.626 7f4387ed8048 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 01/20/2025 00:52:29.516 7f4389248348 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001581 01/20/2025 00:52:29.517 7f4389248348 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 01/20/2025 00:52:29.518 7f4387ed8048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 01/20/2025 00:52:29.518 7f4387ed8048 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 4 01/20/2025 00:52:29.518 7f4387ed8048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 01/20/2025 00:52:29.519 7f4387ed8048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 01/20/2025 00:52:29.519 7f4387ed8048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 00:52:29.519 7f4387ed8048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 01/20/2025 00:52:29.519 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 01/20/2025 00:52:29.520 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 01/20/2025 00:53:04.518 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 00:53:07.393 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737334387393781, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=4), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 00:53:12.685 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7198116, partition_space_available_1k=73892368, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1333252, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10204, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51948, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=5630228, free=3503376, buffers=261884, cached=6780144, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.59, five_min_avg=1.15, fifteen_min_avg=0.61, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 00:54:04.624 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 00:54:13.644 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7271856, partition_space_available_1k=73818628, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1332852, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10372, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51952, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=5695448, free=3364792, buffers=263224, cached=6852168, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.78, five_min_avg=1.31, fifteen_min_avg=0.7, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 00:55:04.547 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 00:55:12.661 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7295592, partition_space_available_1k=73794892, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1337684, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10468, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51964, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=5771776, free=3262732, buffers=265196, cached=6875928, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.34, five_min_avg=1.56, fifteen_min_avg=0.83, cpu_share=9.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 00:55:55.546 7f4387ed8048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 00:56:04.660 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 00:56:12.676 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7749836, partition_space_available_1k=73340648, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1374160, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10612, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51972, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=5861868, free=2686412, buffers=272344, cached=7355008, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.41, five_min_avg=1.94, fifteen_min_avg=1.01, cpu_share=10.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 00:56:45.010 7f4387ed8048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 00:57:04.603 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 00:57:12.674 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7758612, partition_space_available_1k=73331872, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1349592, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10676, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51976, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=5924568, free=2621628, buffers=273752, cached=7355684, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.43, five_min_avg=2.24, fifteen_min_avg=1.17, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 00:58:04.571 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 00:58:12.702 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7759596, partition_space_available_1k=73330888, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1356784, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10676, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51988, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=5940776, free=2604604, buffers=274768, cached=7355484, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.16, five_min_avg=2.08, fifteen_min_avg=1.18, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 00:59:04.553 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 00:59:13.700 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7770768, partition_space_available_1k=73319716, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1361884, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10588, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52028, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=5989008, free=2537656, buffers=275500, cached=7373468, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.19, five_min_avg=2.13, fifteen_min_avg=1.26, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:00:04.583 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:00:12.711 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7782428, partition_space_available_1k=73308056, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1367640, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10628, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52040, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6010020, free=2509372, buffers=276228, cached=7380012, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.34, five_min_avg=2.16, fifteen_min_avg=1.32, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:01:04.592 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:01:13.700 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7793296, partition_space_available_1k=73297188, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1374620, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10644, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52040, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6056956, free=2449468, buffers=276592, cached=7392616, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.02, five_min_avg=2.1, fifteen_min_avg=1.36, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:02:04.597 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:02:12.716 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7798976, partition_space_available_1k=73291508, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1377512, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10712, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52048, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6068276, free=2426552, buffers=276904, cached=7403900, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.91, five_min_avg=2.04, fifteen_min_avg=1.38, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:03:04.611 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:03:12.738 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7811360, partition_space_available_1k=73279124, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1383976, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10752, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52048, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6109856, free=2377316, buffers=277148, cached=7411312, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.96, five_min_avg=2.09, fifteen_min_avg=1.44, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:04:04.592 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:04:12.728 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7823336, partition_space_available_1k=73267148, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1388408, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10760, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52052, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6149520, free=2324932, buffers=277300, cached=7423880, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.09, five_min_avg=2.08, fifteen_min_avg=1.48, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:05:04.604 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:05:12.751 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7825916, partition_space_available_1k=73264568, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1392620, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10676, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52060, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6171652, free=2289532, buffers=277440, cached=7437008, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.99, five_min_avg=2.03, fifteen_min_avg=1.5, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:06:04.635 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:06:13.729 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7839924, partition_space_available_1k=73250560, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1396960, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10696, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52060, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6230408, free=2220680, buffers=277524, cached=7447020, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.93, five_min_avg=1.97, fifteen_min_avg=1.51, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:07:04.695 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:07:13.761 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7846584, partition_space_available_1k=73243900, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1412924, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10728, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52064, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6245808, free=2192056, buffers=277580, cached=7460188, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.93, five_min_avg=1.95, fifteen_min_avg=1.54, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:08:04.637 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:08:13.760 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7858568, partition_space_available_1k=73231916, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1407512, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10760, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52068, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6265396, free=2165828, buffers=277648, cached=7466760, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.36, five_min_avg=2.02, fifteen_min_avg=1.58, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:09:04.663 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:09:13.745 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7869704, partition_space_available_1k=73220780, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1414580, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10764, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52068, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6275600, free=2148852, buffers=277668, cached=7473512, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.26, five_min_avg=2.04, fifteen_min_avg=1.61, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:10:04.647 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:10:13.756 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7871432, partition_space_available_1k=73219052, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1419868, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10832, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52084, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6280952, free=2136996, buffers=277672, cached=7480012, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.5, five_min_avg=2.11, fifteen_min_avg=1.67, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:11:04.635 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:11:12.756 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7863140, partition_space_available_1k=73227344, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1426136, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10792, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52072, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6282228, free=2144668, buffers=277684, cached=7471052, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.12, five_min_avg=2.09, fifteen_min_avg=1.69, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:12:04.712 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:12:12.768 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7869320, partition_space_available_1k=73221164, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1431960, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10816, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52080, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6291028, free=2124044, buffers=277704, cached=7482856, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.09, five_min_avg=2.06, fifteen_min_avg=1.7, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:13:04.648 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:13:13.774 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7881196, partition_space_available_1k=73209288, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1447740, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10844, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52084, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6297148, free=2115556, buffers=277712, cached=7485216, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.93, five_min_avg=2.02, fifteen_min_avg=1.71, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:14:04.649 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:14:13.768 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7881984, partition_space_available_1k=73208500, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1441692, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10876, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52088, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6305976, free=2095988, buffers=277720, cached=7495948, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.89, five_min_avg=1.98, fifteen_min_avg=1.72, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:15:04.681 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:15:13.777 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7892528, partition_space_available_1k=73197956, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1447512, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10876, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52084, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6313644, free=2087244, buffers=277736, cached=7497008, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.81, five_min_avg=1.98, fifteen_min_avg=1.74, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:16:04.653 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:16:13.798 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7898272, partition_space_available_1k=73192212, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1451680, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10932, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52092, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6319472, free=2070168, buffers=277744, cached=7508248, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.57, five_min_avg=2.16, fifteen_min_avg=1.82, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:17:04.682 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:17:12.774 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7909556, partition_space_available_1k=73180928, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1454228, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10888, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52096, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6339256, free=2039424, buffers=277748, cached=7519204, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.56, five_min_avg=2.26, fifteen_min_avg=1.88, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:18:04.653 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:18:12.781 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7915648, partition_space_available_1k=73174836, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1465344, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10912, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52104, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6325592, free=2052100, buffers=277748, cached=7520192, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.32, five_min_avg=2.22, fifteen_min_avg=1.89, cpu_share=8.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:19:04.666 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:19:12.770 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7920500, partition_space_available_1k=73169984, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1464852, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10916, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52108, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6350584, free=2017424, buffers=277752, cached=7529872, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.97, five_min_avg=2.13, fifteen_min_avg=1.88, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:20:04.668 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:20:12.760 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7925712, partition_space_available_1k=73164772, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1467592, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10960, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52100, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6346472, free=2011148, buffers=277764, cached=7540248, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.9, five_min_avg=2.1, fifteen_min_avg=1.89, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:21:04.676 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:21:12.764 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7932884, partition_space_available_1k=73157600, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1473480, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10984, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52104, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6348736, free=2006444, buffers=277776, cached=7542676, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.81, five_min_avg=2.06, fifteen_min_avg=1.89, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:22:04.678 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:22:13.788 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7939700, partition_space_available_1k=73150784, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1478832, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11024, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52108, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6354740, free=1996128, buffers=277776, cached=7546988, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.52, five_min_avg=1.92, fifteen_min_avg=1.85, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:23:04.660 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:23:12.800 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7945992, partition_space_available_1k=73144492, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1483824, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10912, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52116, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6357520, free=1984212, buffers=277788, cached=7556112, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.7, five_min_avg=1.91, fifteen_min_avg=1.85, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:24:04.674 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:24:12.798 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7956344, partition_space_available_1k=73134140, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1489216, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10920, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52116, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6365036, free=1966176, buffers=277796, cached=7566624, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.0, five_min_avg=1.95, fifteen_min_avg=1.87, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:25:04.698 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:25:12.805 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7962792, partition_space_available_1k=73127692, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1495400, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10956, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52428, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6363704, free=1965740, buffers=277796, cached=7568392, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.73, five_min_avg=1.87, fifteen_min_avg=1.85, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:26:04.698 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:26:12.819 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7968660, partition_space_available_1k=73121824, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1500344, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10964, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52424, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6374088, free=1949544, buffers=277796, cached=7574204, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.0, five_min_avg=1.91, fifteen_min_avg=1.86, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:27:04.704 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:27:13.825 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7974520, partition_space_available_1k=73115964, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1506036, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10984, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52428, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6380708, free=1934884, buffers=277800, cached=7582240, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.79, five_min_avg=1.88, fifteen_min_avg=1.85, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:28:04.709 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:28:12.843 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7980448, partition_space_available_1k=73110036, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1510864, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10984, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52436, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6382156, free=1923900, buffers=277800, cached=7591776, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.81, five_min_avg=1.86, fifteen_min_avg=1.85, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:29:04.720 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:29:12.852 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7990836, partition_space_available_1k=73099648, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1514856, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10936, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52436, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6387436, free=1917952, buffers=277800, cached=7592444, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.85, five_min_avg=1.87, fifteen_min_avg=1.85, cpu_share=8.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:30:04.778 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:30:12.859 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7991780, partition_space_available_1k=73098704, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1521320, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10952, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52440, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6388276, free=1906456, buffers=277800, cached=7603100, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.93, five_min_avg=1.89, fifteen_min_avg=1.86, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:31:04.760 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:31:07.867 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737336667867025, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=4)], _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:31:13.870 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8013728, partition_space_available_1k=73076756, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1497052, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10980, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52444, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6349552, free=1926596, buffers=277800, cached=7621684, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.58, five_min_avg=1.81, fifteen_min_avg=1.84, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:32:04.763 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:32:12.875 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8018636, partition_space_available_1k=73071848, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1499196, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10996, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52448, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6353652, free=1912152, buffers=277804, cached=7632024, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.54, five_min_avg=1.76, fifteen_min_avg=1.81, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:33:04.792 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:33:12.890 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8029956, partition_space_available_1k=73060528, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1503404, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11044, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52448, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6354332, free=1910996, buffers=277812, cached=7632492, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.75, five_min_avg=1.77, fifteen_min_avg=1.82, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:34:04.770 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:34:12.880 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8031664, partition_space_available_1k=73058820, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1509116, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11044, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52452, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6330480, free=1922744, buffers=277816, cached=7644592, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.69, five_min_avg=1.75, fifteen_min_avg=1.81, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:35:04.769 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:35:12.888 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8039640, partition_space_available_1k=73050844, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1516396, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10940, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52456, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6361656, free=1883432, buffers=277820, cached=7652724, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.01, five_min_avg=1.83, fifteen_min_avg=1.83, cpu_share=8.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:36:04.800 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:36:13.899 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8045944, partition_space_available_1k=73044540, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1519280, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10960, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52460, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6369940, free=1872856, buffers=277820, cached=7655016, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.44, five_min_avg=1.96, fifteen_min_avg=1.87, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:37:04.798 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:37:12.899 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8051288, partition_space_available_1k=73039196, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1525480, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10964, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52460, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6368836, free=1863324, buffers=277820, cached=7665652, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.81, five_min_avg=1.9, fifteen_min_avg=1.86, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:38:04.797 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:38:12.918 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8058224, partition_space_available_1k=73032260, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1530072, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10972, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52464, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6370228, free=1856012, buffers=277824, cached=7671568, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.73, five_min_avg=1.9, fifteen_min_avg=1.87, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:39:04.799 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:39:12.920 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8063744, partition_space_available_1k=73026740, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1534844, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11008, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52468, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6356092, free=1859744, buffers=277824, cached=7681972, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.82, five_min_avg=1.89, fifteen_min_avg=1.87, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:40:04.828 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:40:12.927 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8074308, partition_space_available_1k=73016176, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1538272, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11036, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52468, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6378492, free=1836268, buffers=277824, cached=7683048, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.47, five_min_avg=1.79, fifteen_min_avg=1.83, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:41:04.819 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:41:12.926 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8074844, partition_space_available_1k=73015640, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1542932, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10968, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52472, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6375552, free=1828304, buffers=277824, cached=7693952, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.79, five_min_avg=1.8, fifteen_min_avg=1.84, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:42:04.846 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:42:12.939 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8085672, partition_space_available_1k=73004812, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1548048, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10976, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52476, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6382404, free=1814640, buffers=277824, cached=7700764, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.43, five_min_avg=2.02, fifteen_min_avg=1.91, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:43:04.826 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:43:12.944 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8092436, partition_space_available_1k=72998048, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1554864, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11000, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52480, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6390012, free=1800876, buffers=277824, cached=7706920, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.28, five_min_avg=2.04, fifteen_min_avg=1.92, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:44:04.827 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:44:12.938 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8098436, partition_space_available_1k=72992048, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1561272, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11012, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52484, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6392256, free=1792632, buffers=277824, cached=7712920, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.0, five_min_avg=1.99, fifteen_min_avg=1.91, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:45:04.849 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:45:12.939 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8104720, partition_space_available_1k=72985764, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1564592, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11012, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52484, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6396088, free=1782044, buffers=277824, cached=7719676, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.04, five_min_avg=1.99, fifteen_min_avg=1.92, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:46:04.849 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:46:12.942 7f438e5cedf0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8110044, partition_space_available_1k=72980440, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1566848, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11016, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52484, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6401136, free=1767624, buffers=277824, cached=7729048, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.83, five_min_avg=1.92, fifteen_min_avg=1.9, cpu_share=8.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:47:04.092 7f4389248348 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 01/20/2025 01:47:04.092 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[Established] 01/20/2025 01:47:04.098 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 01/20/2025 01:47:04.098 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[Established] -> State[Connect] 01/20/2025 01:47:04.099 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:47:04.099 7f4388c3ee48 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Established to Connect on EvTcpClose >>, ] description = Collector:Collector[Established to Connect on EvTcpClose] connection down >>, ] >> 01/20/2025 01:47:04.099 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:47:04.099 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 01/20/2025 01:47:04.099 7f4388c3ee48 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 01/20/2025 01:47:04.101 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 01/20/2025 01:47:04.102 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:47:04.102 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:47:04.171 7f4389248348 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001017 01/20/2025 01:47:04.172 7f4389248348 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 01/20/2025 01:47:04.172 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 01/20/2025 01:47:04.173 7f4388c3ee48 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 01/20/2025 01:47:04.173 7f4388c3ee48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 01/20/2025 01:47:04.173 7f4388c3ee48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 01/20/2025 01:47:04.173 7f4388c3ee48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 01:47:04.173 7f4388c3ee48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 01/20/2025 01:47:04.173 7f4388c3ee48 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 01/20/2025 01:47:04.174 7f4387ed8048 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 01/20/2025 01:47:04.175 7f4387ed8048 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 01/20/2025 01:47:18.541 7f1fb22def20 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 01/20/2025 01:47:18.541 7f1fb22def20 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 01/20/2025 01:47:18.541 7f1fb22def20 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 01/20/2025 01:47:18.572 7f1fac6fe748 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 01/20/2025 01:47:18.572 7f1fac6fe748 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 01/20/2025 01:47:18.573 7f1fab314048 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 01/20/2025 01:47:18.573 7f1fab314048 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 01:47:18.573 7f1fab314048 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 01/20/2025 01:47:18.574 7f1fab314048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 01/20/2025 01:47:18.574 7f1fab314048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:47:18.574 7f1fab314048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/20/2025 01:47:18.574 7f1fab314048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:47:18.574 7f1fab314048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:47:18.574 7f1fab314048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 01/20/2025 01:47:18.574 7f1fab314048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:47:21.359 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 94537270109468223081139197337835364206539177810879441038868846995004006080747, 'start': '1737337636000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 110149863120647927509892561123495881608747483564226840696695047398324932250252, 'start': '1737337637000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 27691419639454453072800012030658330408095100336004145304229453583761079062662, 'start': '1737337636000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 01/20/2025 01:47:21.785 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:47:21.818 7f1fab314448 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-01-16 01:14:15.868302", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7545-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = nightly.r24.1.centos-nightly_r24.1_centos.el7 installed_package_version = nightly.r24.1.centos-nightly_r24.1_centos.el7 >> 01/20/2025 01:47:21.818 7f1fab314448 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:47:21.819 7f1fb22def20 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737337641819084, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:47:21.819 7f1fb22def20 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737337641819523, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737337636000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737337637000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737337636000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-01-16 01:14:15.868302", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7545-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:47:21.822 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:47:21.822 7f1fab314448 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/20/2025 01:47:21.823 7f1fab314448 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:47:21.823 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:47:21.823 7f1fab314448 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737337636000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737337637000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737337636000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-01-16 01:14:15.868302", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7545-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} >> 01/20/2025 01:47:21.823 7f1fab314448 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:47:22.579 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:47:22.615 7f1fab314448 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 01/20/2025 01:47:22.615 7f1fab314448 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:47:22.615 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:47:22.615 7f1fab314448 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:47:22.615 7f1fab314448 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:47:22.616 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 01/20/2025 01:47:22.616 7f1fab314448 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 01/20/2025 01:47:22.618 7f1fab314448 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 01/20/2025 01:47:22.618 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:47:22.618 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:47:23.465 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:47:25.372 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 01/20/2025 01:47:25.374 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:47:27.885 7f1fb22def20 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 01/20/2025 01:47:27.885 7f1fb22def20 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737337647885434, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.22:8086'], status='Initializing', description='Connect to ClientInit on EvTcpConnected')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[Connect to ClientInit on EvTcpConnected] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:47:27.887 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:47:31.042 7f1fb22def20 [contrail-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 2 01/20/2025 01:47:33.872 7f1fab2d2448 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 01/20/2025 01:47:33.873 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 01/20/2025 01:47:33.873 7f1fab314448 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 01/20/2025 01:47:33.873 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:47:33.873 7f1fab314448 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 01/20/2025 01:47:33.873 7f1fab314448 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:47:33.874 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:47:33.874 7f1fab314448 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 01/20/2025 01:47:33.874 7f1fab314448 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:47:37.012 7f1fb22def20 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 01/20/2025 01:47:37.874 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:47:37.876 7f1fab314448 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 01/20/2025 01:47:37.876 7f1fab314448 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:47:37.876 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:47:37.876 7f1fab314448 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:47:37.876 7f1fab314448 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:47:37.876 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 01/20/2025 01:47:37.876 7f1fab314448 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 01/20/2025 01:47:37.877 7f1fab314448 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 01/20/2025 01:47:37.877 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:47:37.877 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:47:37.891 7f1fab2d2448 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001017 01/20/2025 01:47:37.892 7f1fab2d2448 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 01/20/2025 01:47:37.892 7f1fab314448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 01/20/2025 01:47:37.893 7f1fab314448 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 01/20/2025 01:47:37.893 7f1fab314448 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 01/20/2025 01:47:37.893 7f1fab314448 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 01:47:37.893 7f1fab314448 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 01/20/2025 01:47:37.893 7f1fab314448 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 01/20/2025 01:47:37.893 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 01/20/2025 01:47:37.894 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 01/20/2025 01:47:42.044 7f1fb22def20 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737337662044537, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737337636000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737337637000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737337636000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-01-16 01:14:15.868302", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7545-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:47:42.045 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=8174160, partition_space_available_1k=72916324, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1220524, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=8756, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44568, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=4993500, free=3054008, buffers=282528, cached=7845596, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=9.09, five_min_avg=3.7, fifteen_min_avg=2.5, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:48:22.467 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:48:24.787 7f1fb22def20 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737337704787490, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.42:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:48:30.978 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737337710978616, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=3), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:48:30.978 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 01/20/2025 01:48:37.001 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=9219548, partition_space_available_1k=71870936, percentage_partition_space_used=11, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1338388, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=9908, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44592, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=5480464, free=1357160, buffers=353132, cached=8984876, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=12.57, five_min_avg=5.71, fifteen_min_avg=3.26, cpu_share=22.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:49:22.301 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:49:33.986 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10042504, partition_space_available_1k=71047980, percentage_partition_space_used=12, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1342188, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10012, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44612, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=5896112, free=174000, buffers=380228, cached=9725292, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=8.84, five_min_avg=5.79, fifteen_min_avg=3.43, cpu_share=15.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:50:15.037 7f1fab2d2448 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 01/20/2025 01:50:15.039 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[Established] 01/20/2025 01:50:15.040 7f1fab314148 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 01/20/2025 01:50:15.040 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[Established] -> State[Connect] 01/20/2025 01:50:15.041 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:15.041 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Established to Connect on EvTcpClose >>, ] description = Collector:Collector[Established to Connect on EvTcpClose] connection down >>, ] >> 01/20/2025 01:50:15.041 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:15.041 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 01/20/2025 01:50:15.042 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 01/20/2025 01:50:15.042 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 01/20/2025 01:50:15.043 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:50:15.043 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:50:15.149 7f1fab2d2448 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 01/20/2025 01:50:15.149 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 01/20/2025 01:50:15.150 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 01/20/2025 01:50:15.150 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:15.150 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 01/20/2025 01:50:15.151 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:15.151 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:15.151 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 3 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:50:15.151 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:19.152 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:50:19.153 7f1fab314148 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 01/20/2025 01:50:19.153 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:50:19.154 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:19.154 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:50:19.154 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:19.154 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 01/20/2025 01:50:19.155 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 01/20/2025 01:50:19.155 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 01/20/2025 01:50:19.155 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:50:19.155 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:50:19.172 7f1fab2d2448 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 01/20/2025 01:50:19.172 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 01/20/2025 01:50:19.172 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 01/20/2025 01:50:19.173 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:19.173 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 01/20/2025 01:50:19.173 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:19.173 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:19.173 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:50:19.173 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:22.058 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:50:22.301 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:22.301 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. database_usage = [ << disk_space_used_1k = 10441948 disk_space_available_1k = 70648536 analytics_db_size_1k = 29876 >>, ] >> 01/20/2025 01:50:22.301 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:23.173 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:50:23.175 7f1fab314148 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 01/20/2025 01:50:23.175 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:50:23.175 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:23.176 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:50:23.176 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:23.176 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 01/20/2025 01:50:23.176 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 01/20/2025 01:50:23.177 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 01/20/2025 01:50:23.177 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:50:23.177 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:50:28.000 7f1fab2d2448 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 01/20/2025 01:50:28.000 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 01/20/2025 01:50:28.001 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 01/20/2025 01:50:28.001 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:28.001 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 01/20/2025 01:50:28.001 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:28.001 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:28.002 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:50:28.002 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:29.919 7f1fb22def20 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737337829919543, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737337636000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737337637000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737337636000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-01-16 01:14:15.868302", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7545-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:50:29.920 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10442044, partition_space_available_1k=70648440, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=['/var/crashes/core.contrail-collec.1.cn-jenkins-deploy-platform-ansible-os-2650-2.1737337815'], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1343612, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10680, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44784, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=5033900, free=1122252, buffers=384744, cached=9634736, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.51, five_min_avg=5.44, fifteen_min_avg=3.45, cpu_share=11.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:50:29.920 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:29.920 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737337636000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737337637000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737337636000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-01-16 01:14:15.868302", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7545-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} >> 01/20/2025 01:50:29.920 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:29.920 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:29.920 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 10442044 partition_space_available_1k = 70648440 percentage_partition_space_used = 13 >> } all_core_file_list = [ /var/crashes/core.contrail-collec.1.cn-jenkins-deploy-platform-ansible-os-2650-2.1737337815, ] process_mem_cpu_usage = { cassandra : << mem_virt = 1343612 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 10680 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 44784 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175632 used = 5033900 free = 1122252 buffers = 384744 cached = 9634736 node_type = database-node >> system_cpu_usage = << one_min_avg = 5.51 five_min_avg = 5.44 fifteen_min_avg = 3.45 cpu_share = 11.0 node_type = database-node >> >> 01/20/2025 01:50:29.920 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:32.003 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:50:32.004 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:50:32.005 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:50:32.005 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:32.005 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:50:32.005 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:32.005 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:50:32.005 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:50:32.005 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:32.006 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:50:32.006 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:32.006 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:32.006 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:50:32.006 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:36.005 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:50:36.005 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:50:36.006 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:50:36.006 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:36.006 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:50:36.006 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:36.006 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:50:36.006 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:50:36.006 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:36.007 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:50:36.007 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:36.007 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:36.007 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:50:36.007 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:40.007 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:50:40.009 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:50:40.009 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:50:40.009 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:40.010 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:50:40.010 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:40.010 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:50:40.011 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:50:40.011 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:40.011 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:50:40.011 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:40.012 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:40.012 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:50:40.012 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:44.012 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:50:44.014 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:50:44.014 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:50:44.014 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:44.014 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:50:44.014 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:44.014 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:50:44.015 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:50:44.015 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:44.015 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:50:44.015 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:44.015 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:44.015 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:50:44.015 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:48.018 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:50:48.019 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:50:48.019 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:50:48.019 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:48.019 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:50:48.019 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:48.020 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:50:48.020 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:50:48.020 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:48.020 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:50:48.020 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:48.020 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:48.020 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:50:48.020 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:52.021 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:50:52.023 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:50:52.023 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:50:52.023 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:52.023 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:50:52.023 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:52.024 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:50:52.024 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:50:52.024 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:52.024 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:50:52.024 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:52.024 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:52.024 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:50:52.024 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:56.023 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:50:56.025 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:50:56.025 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:50:56.025 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:56.025 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:50:56.025 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:50:56.025 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:50:56.026 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:50:56.026 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:56.026 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:50:56.026 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:56.026 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:50:56.026 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:50:56.026 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:00.026 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:00.027 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:00.027 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:00.028 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:00.028 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:00.028 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:00.028 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:00.028 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:00.028 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:00.028 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:00.029 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:00.029 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:00.029 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:00.029 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:04.029 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:04.031 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:04.031 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:04.031 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:04.031 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:04.031 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:04.032 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:04.032 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:04.032 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:04.032 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:04.032 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:04.032 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:04.032 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:04.032 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:08.035 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:08.036 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:08.036 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:08.036 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:08.037 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:08.037 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:08.037 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:08.037 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:08.037 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:08.037 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:08.037 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:08.037 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:08.037 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:08.037 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:12.038 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:12.040 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:12.041 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:12.041 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:12.041 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:12.041 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:12.041 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:12.041 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:12.041 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:12.042 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:12.042 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:12.042 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:12.042 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:12.042 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:16.044 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:16.045 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:16.046 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:16.046 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:16.046 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:16.046 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:16.046 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:16.046 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:16.047 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:16.047 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:16.047 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:16.047 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:16.047 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:16.047 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:20.048 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:20.050 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:20.050 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:20.050 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:20.050 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:20.051 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:20.051 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:20.051 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:20.051 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:20.051 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:20.052 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:20.052 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:20.052 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:20.052 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:22.027 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:51:22.240 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:22.240 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. database_usage = [ << disk_space_used_1k = 10333160 disk_space_available_1k = 70757324 analytics_db_size_1k = 29876 >>, ] >> 01/20/2025 01:51:22.240 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:24.050 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:24.051 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:24.051 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:24.051 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:24.051 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:24.051 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:24.052 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:24.052 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:24.052 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:24.052 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:24.052 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:24.052 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:24.052 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:24.052 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:28.051 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:28.053 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:28.054 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:28.054 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:28.054 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:28.054 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:28.054 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:28.054 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:28.054 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:28.055 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:28.055 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:28.055 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:28.055 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:28.055 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:29.903 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10333204, partition_space_available_1k=70757280, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1346856, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11096, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44868, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=4254860, free=1950960, buffers=388784, cached=9581028, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.39, five_min_avg=5.04, fifteen_min_avg=3.43, cpu_share=3.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:51:29.904 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:29.904 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 10333204 partition_space_available_1k = 70757280 percentage_partition_space_used = 13 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1346856 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 11096 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 44868 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175632 used = 4254860 free = 1950960 buffers = 388784 cached = 9581028 node_type = database-node >> system_cpu_usage = << one_min_avg = 4.39 five_min_avg = 5.04 fifteen_min_avg = 3.43 cpu_share = 3.75 node_type = database-node >> >> 01/20/2025 01:51:29.904 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:32.055 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:32.056 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:32.056 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:32.056 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:32.057 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:32.057 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:32.057 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:32.057 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:32.057 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:32.058 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:32.058 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:32.058 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:32.058 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:32.058 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:36.060 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:36.061 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:36.061 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:36.061 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:36.061 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:36.061 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:36.061 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:36.061 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:36.062 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:36.062 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:36.062 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:36.062 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:36.062 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:36.062 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:40.063 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:40.065 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:40.065 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:40.065 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:40.065 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:40.065 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:40.065 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:40.066 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:40.066 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:40.066 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:40.066 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:40.066 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:40.066 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:40.066 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:44.067 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:44.069 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:44.069 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:44.069 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:44.069 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:44.069 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:44.069 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:44.070 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:44.070 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:44.070 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:44.070 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:44.070 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:44.070 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:44.070 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:48.072 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:48.073 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:48.073 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:48.073 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:48.073 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:48.073 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:48.073 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:48.074 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:48.074 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:48.074 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:48.074 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:48.074 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:48.074 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:48.074 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:52.076 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:52.078 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:52.078 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:52.078 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:52.078 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:52.078 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:52.078 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:52.079 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:52.079 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:52.079 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:52.079 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:52.079 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:52.079 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:52.079 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:56.079 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:51:56.080 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:51:56.080 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:51:56.080 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:56.080 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:51:56.080 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:51:56.080 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:51:56.081 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:51:56.081 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:56.081 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:51:56.081 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:56.081 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:51:56.081 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:51:56.081 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:00.084 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:52:00.085 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:52:00.085 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:52:00.085 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:00.085 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:52:00.085 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:00.085 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:52:00.086 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:52:00.086 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:00.086 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:52:00.086 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:00.086 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:00.086 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:52:00.086 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:04.087 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:52:04.089 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:52:04.089 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:52:04.089 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:04.089 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:52:04.089 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:04.089 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:52:04.090 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:52:04.090 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:04.090 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:52:04.090 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:04.090 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:04.090 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:52:04.090 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:08.092 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:52:08.093 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:52:08.093 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:52:08.093 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:08.094 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:52:08.094 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:08.094 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:52:08.094 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:52:08.094 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:08.094 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:52:08.094 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:08.094 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:08.095 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:52:08.095 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:12.094 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:52:12.095 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:52:12.095 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:52:12.095 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:12.095 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:52:12.096 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:12.096 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:52:12.096 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:52:12.096 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:12.096 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:52:12.096 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:12.096 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:12.097 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:52:12.097 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:16.099 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:52:16.100 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:52:16.100 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:52:16.100 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:16.101 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:52:16.101 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:16.101 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:52:16.101 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:52:16.101 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:16.101 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:52:16.102 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:16.102 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:16.102 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.42:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:52:16.102 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:20.101 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:52:20.103 7f1fab314148 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 01/20/2025 01:52:20.103 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:52:20.103 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:20.103 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:52:20.103 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:20.103 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 01/20/2025 01:52:20.104 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 01/20/2025 01:52:20.104 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:20.104 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.49:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 01/20/2025 01:52:20.104 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:20.104 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:20.104 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 5 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.49:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:52:20.104 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:22.078 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:52:22.339 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:22.339 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. database_usage = [ << disk_space_used_1k = 10371432 disk_space_available_1k = 70719052 analytics_db_size_1k = 29876 >>, ] >> 01/20/2025 01:52:22.339 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:24.104 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:52:24.105 7f1fab314148 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 01/20/2025 01:52:24.105 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:52:24.105 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:24.105 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:52:24.105 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:24.105 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 01/20/2025 01:52:24.106 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 01/20/2025 01:52:24.106 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 01/20/2025 01:52:24.106 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:52:24.106 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:52:26.429 7f1fab2d2448 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 01/20/2025 01:52:26.430 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 01/20/2025 01:52:26.430 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 01/20/2025 01:52:26.430 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:26.431 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 01/20/2025 01:52:26.431 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:26.431 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:26.431 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 6 pid = 1 http_port = 8103 start_time = 1737337638573893 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.22:8086, 10.0.0.42:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 5 >> 01/20/2025 01:52:26.431 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:52:30.430 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:52:30.433 7f1fab314148 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 01/20/2025 01:52:30.434 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:52:30.434 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:30.434 7f1fab314148 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:52:30.434 7f1fab314148 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:52:30.434 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 01/20/2025 01:52:30.435 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 01/20/2025 01:52:30.435 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 01/20/2025 01:52:30.435 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:52:30.435 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:52:30.456 7f1fab2d2448 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001016 01/20/2025 01:52:30.457 7f1fab2d2448 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 01/20/2025 01:52:30.458 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 01/20/2025 01:52:30.459 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 01/20/2025 01:52:30.459 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 01/20/2025 01:52:30.459 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 01/20/2025 01:52:30.459 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 01:52:30.460 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 01/20/2025 01:52:30.460 7f1fab314148 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 01/20/2025 01:52:30.460 7f1fab314448 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 01/20/2025 01:52:30.462 7f1fab314448 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 01/20/2025 01:52:31.925 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10387228, partition_space_available_1k=70703256, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1347912, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11488, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=44984, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=5320564, free=741856, buffers=393932, cached=9719280, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.5, five_min_avg=4.39, fifteen_min_avg=3.32, cpu_share=8.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:52:36.729 7f1fab2d2448 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001582 01/20/2025 01:52:36.729 7f1fab2d2448 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 01/20/2025 01:52:36.730 7f1fab314148 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 01/20/2025 01:52:36.731 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 4 01/20/2025 01:52:36.731 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 01/20/2025 01:52:36.732 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 01/20/2025 01:52:36.732 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 01:52:36.735 7f1fab314148 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 01/20/2025 01:52:36.735 7f1fab314448 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 01/20/2025 01:52:36.736 7f1fab314448 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 01/20/2025 01:53:22.120 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:53:30.924 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10501416, partition_space_available_1k=70589068, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1390108, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11628, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=45068, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6047940, free=405552, buffers=397324, cached=9324816, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.07, five_min_avg=4.0, fifteen_min_avg=3.25, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:54:22.127 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:54:30.794 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10936620, partition_space_available_1k=70153864, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1365212, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10988, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=45096, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6025072, free=326520, buffers=407604, cached=9416436, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.65, five_min_avg=3.54, fifteen_min_avg=3.14, cpu_share=9.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:55:22.153 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:55:30.809 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10940928, partition_space_available_1k=70149556, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1391388, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11036, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=45096, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6034156, free=313976, buffers=407628, cached=9419872, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.01, five_min_avg=3.34, fifteen_min_avg=3.09, cpu_share=6.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:56:22.126 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:56:30.925 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=10941096, partition_space_available_1k=70149388, percentage_partition_space_used=13, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1379776, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11884, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=45100, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6065280, free=322416, buffers=406816, cached=9381120, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.79, five_min_avg=3.08, fifteen_min_avg=3.02, cpu_share=6.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:57:22.088 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:57:30.201 7f1fb22def20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=11725684, partition_space_available_1k=69364800, percentage_partition_space_used=14, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1392120, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12152, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=45144, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=3845968, free=2578860, buffers=417672, cached=9333132, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.33, five_min_avg=2.96, fifteen_min_avg=2.98, cpu_share=9.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:57:58.356 7f1fab2d2248 [contrail-database-nodemgr] [DEBUG]: DBG: event_process_state:contrail-query-engine,group:contrail-query-engine,state:PROCESS_STATE_EXITED 01/20/2025 01:57:58.356 7f1fab2d2248 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737338278356770, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737337636000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_EXITED', start_count=1, stop_count=0, exit_count=1, last_start_time='1737337637000000', last_stop_time='', last_exit_time='1737338278356723', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737337636000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-01-16 01:14:15.868302", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7545-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:58:25.295 7f00f9cfebc8 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 01/20/2025 01:58:25.295 7f00f9cfebc8 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 01/20/2025 01:58:25.295 7f00f9cfebc8 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 01/20/2025 01:58:25.305 7f00f7648a20 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 01/20/2025 01:58:25.305 7f00f7648a20 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 01/20/2025 01:58:25.306 7f00f764c390 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 01/20/2025 01:58:25.307 7f00f764c390 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 01:58:25.307 7f00f764c390 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 01/20/2025 01:58:25.307 7f00f764c390 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 01/20/2025 01:58:25.307 7f00f764c390 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:58:25.308 7f00f764c390 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/20/2025 01:58:25.308 7f00f764c390 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:58:25.308 7f00f764c390 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:58:25.308 7f00f764c390 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1737338305307505 collector_name = collector_ip = collector_list = [ 10.0.0.42:8086, 10.0.0.22:8086, 10.0.0.49:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 01/20/2025 01:58:25.308 7f00f764c390 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:58:25.407 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 19823907084866563575028582132929432231459775043666008125907604444084252875445, 'start': '1737338303000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 97466166093645153907488851742003864297059118161072044403225965015241535695136, 'start': '1737338303000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 24696692466199417206173843772690024526697403869600089221756841918176705470029, 'start': '1737338303000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 01/20/2025 01:58:25.545 7f00fff5ea20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:58:25.546 7f00fff5ea20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-01-20 00:44:39.080355", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7550-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 01/20/2025 01:58:25.546 7f00fff5ea20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:58:25.547 7f00f9cfebc8 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737338305547640, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:58:25.548 7f00f9cfebc8 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737338305547995, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737338303000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737338303000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737338303000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-01-20 00:44:39.080355", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7550-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:58:25.549 7f00f7706588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:58:25.549 7f00f7706588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/20/2025 01:58:25.549 7f00f7706588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:58:25.550 7f00f7706588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:58:25.550 7f00f7706588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737338303000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737338303000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1737338303000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-01-20 00:44:39.080355", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7550-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 01/20/2025 01:58:25.550 7f00f7706588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:58:25.857 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:58:26.141 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 01/20/2025 01:58:26.143 7f00f79bc588 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:58:26.143 7f00f79bc588 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. database_usage = [ << disk_space_used_1k = 12140532 disk_space_available_1k = 68949952 analytics_db_size_1k = 42900 >>, ] >> 01/20/2025 01:58:26.143 7f00f79bc588 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:58:26.586 7f00f9cfebc8 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 01/20/2025 01:58:26.586 7f00f9cfebc8 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737338306586909, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:58:26.588 7f00f7618eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:58:26.589 7f00f7618eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 01/20/2025 01:58:26.589 7f00f7618eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 01/20/2025 01:58:29.307 7f00f762ff60 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 01/20/2025 01:58:29.309 7f00f762ff60 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 01/20/2025 01:58:29.310 7f00f762ff60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 01/20/2025 01:58:29.310 7f00f762ff60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:58:29.310 7f00f762ff60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2650-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 01/20/2025 01:58:29.310 7f00f762ff60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 01/20/2025 01:58:29.310 7f00f762ff60 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 01/20/2025 01:58:29.310 7f00f762ff60 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 01/20/2025 01:58:29.311 7f00f762ff60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 01/20/2025 01:58:29.312 7f00f762ff60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:58:29.312 7f00f762ff60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 01/20/2025 01:58:29.316 7f00f762f0f0 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001017 01/20/2025 01:58:29.317 7f00f762f0f0 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 01/20/2025 01:58:29.317 7f00f58baa20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 01/20/2025 01:58:29.317 7f00f58baa20 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 01/20/2025 01:58:29.318 7f00f58baa20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 01/20/2025 01:58:29.318 7f00f58baa20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 01:58:29.318 7f00f58baa20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 01/20/2025 01:58:29.319 7f00f58baa20 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 01/20/2025 01:58:29.319 7f00f58ba2e8 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 01/20/2025 01:58:29.320 7f00f58ba2e8 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 01/20/2025 01:58:29.721 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737338309721786, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=1, pending=1, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:58:29.722 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 01/20/2025 01:58:31.689 7f00f9cfebc8 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 2 01/20/2025 01:58:36.878 7f00f9cfebc8 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737338316878424, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737338303000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737338303000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1737338303000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-01-20 00:44:39.080355", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7550-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:58:36.879 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12143180, partition_space_available_1k=68947304, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=['/var/crashes/core.contrail-collec.1.cn-jenkins-deploy-platform-ansible-os-2650-2.1737337815'], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1298560, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=9700, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51372, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=5518708, free=548188, buffers=432108, cached=9676628, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.35, five_min_avg=3.22, fifteen_min_avg=3.08, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 01:59:25.788 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 01:59:26.711 7f00f9cfebc8 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737338366711502, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.42:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:59:28.672 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2650-2.', _node_type='Database', _instance_id='0', _timestamp=1737338368672728, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 01/20/2025 01:59:33.885 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12144644, partition_space_available_1k=68945840, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1306520, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=9868, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51652, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=5590560, free=472508, buffers=432128, cached=9680436, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.79, five_min_avg=2.79, fifteen_min_avg=2.93, cpu_share=7.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:00:25.785 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:00:33.894 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12151124, partition_space_available_1k=68939360, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1320176, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10108, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51596, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=5979476, free=283040, buffers=431536, cached=9481580, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.06, five_min_avg=2.65, fifteen_min_avg=2.87, cpu_share=7.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:01:25.900 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:01:33.910 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12154132, partition_space_available_1k=68936352, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1333392, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10252, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51596, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6011008, free=273260, buffers=431512, cached=9459852, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.02, five_min_avg=2.49, fifteen_min_avg=2.8, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:02:25.858 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:02:33.925 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12158084, partition_space_available_1k=68932400, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1335568, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10332, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51596, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6018364, free=278100, buffers=431592, cached=9447576, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.62, five_min_avg=2.28, fifteen_min_avg=2.71, cpu_share=6.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:03:25.827 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:03:33.923 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12164576, partition_space_available_1k=68925908, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1374056, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10412, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51592, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6022936, free=287484, buffers=431272, cached=9433940, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.0, five_min_avg=2.22, fifteen_min_avg=2.66, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:04:25.840 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:04:33.937 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12169876, partition_space_available_1k=68920608, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1352904, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10416, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51596, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6032868, free=294136, buffers=431068, cached=9417560, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.54, five_min_avg=2.03, fifteen_min_avg=2.56, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:05:25.844 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:05:33.936 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12168708, partition_space_available_1k=68921776, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1358548, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10468, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51592, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6036588, free=303032, buffers=430636, cached=9405376, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.51, five_min_avg=1.92, fifteen_min_avg=2.49, cpu_share=5.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:06:25.868 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:06:33.952 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12172948, partition_space_available_1k=68917536, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1365184, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10584, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51712, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6033456, free=311064, buffers=430936, cached=9400176, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.19, five_min_avg=1.75, fifteen_min_avg=2.39, cpu_share=5.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:07:25.857 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:07:33.961 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12176716, partition_space_available_1k=68913768, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1389212, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10616, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51708, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6057576, free=310236, buffers=430952, cached=9376868, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.3, five_min_avg=1.68, fifteen_min_avg=2.33, cpu_share=6.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:08:25.904 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:08:33.962 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12180264, partition_space_available_1k=68910220, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1389576, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10656, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51708, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6059548, free=303388, buffers=430952, cached=9381744, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.1, five_min_avg=1.55, fifteen_min_avg=2.24, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:09:25.901 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:09:33.971 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12185652, partition_space_available_1k=68904832, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1383608, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10688, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51708, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6056736, free=316316, buffers=430948, cached=9371632, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.36, five_min_avg=1.53, fifteen_min_avg=2.18, cpu_share=6.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:10:25.900 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:10:33.978 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12185084, partition_space_available_1k=68905400, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1391588, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10676, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51708, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6082280, free=310576, buffers=430936, cached=9351840, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.34, five_min_avg=1.48, fifteen_min_avg=2.13, cpu_share=6.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:11:25.910 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:11:33.979 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12188376, partition_space_available_1k=68902108, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1395212, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10704, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51704, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6097596, free=285520, buffers=431072, cached=9361444, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.36, five_min_avg=1.45, fifteen_min_avg=2.07, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:12:26.036 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:12:34.003 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12192268, partition_space_available_1k=68898216, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1402824, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10712, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51712, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6084524, free=317804, buffers=430936, cached=9342368, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.09, five_min_avg=1.35, fifteen_min_avg=2.0, cpu_share=6.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:13:25.940 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:13:34.000 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12196000, partition_space_available_1k=68894484, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1407464, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10772, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51712, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6107492, free=308892, buffers=430928, cached=9328320, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.08, five_min_avg=1.3, fifteen_min_avg=1.94, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:13:50.363 7f00f58ab048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 02:14:24.329 7f00f9ad96d8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 02:14:25.902 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:14:33.997 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12555120, partition_space_available_1k=68535364, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1412056, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10940, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51620, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6101592, free=359704, buffers=435492, cached=9278844, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.12, five_min_avg=1.26, fifteen_min_avg=1.89, cpu_share=7.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:14:43.113 7f00f96c72e8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 02:15:18.519 7f00f77064e0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 02:15:25.933 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:15:33.995 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12559204, partition_space_available_1k=68531280, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1418424, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10976, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51632, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6099664, free=356204, buffers=435736, cached=9284028, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.18, five_min_avg=1.23, fifteen_min_avg=1.83, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:15:44.092 7f00f7a3c6d8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 02:16:00.856 7f00f758eeb8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 02:16:25.924 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:16:34.006 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12561252, partition_space_available_1k=68529232, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1425576, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10804, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51648, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6111740, free=388216, buffers=435440, cached=9240236, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.59, five_min_avg=1.33, fifteen_min_avg=1.83, cpu_share=6.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:17:25.922 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:17:33.807 7f00f75c0780 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 02:17:34.018 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12564840, partition_space_available_1k=68525644, percentage_partition_space_used=15, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1427748, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10816, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51652, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6131896, free=358968, buffers=435544, cached=9249224, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.36, five_min_avg=1.31, fifteen_min_avg=1.79, cpu_share=6.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:18:25.967 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:18:34.014 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12574504, partition_space_available_1k=68515980, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1453632, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12608, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51648, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6174580, free=326124, buffers=435212, cached=9239716, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.75, five_min_avg=1.77, fifteen_min_avg=1.92, cpu_share=8.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:19:25.923 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:19:35.017 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12586008, partition_space_available_1k=68504476, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1482224, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12636, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51644, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6195040, free=309828, buffers=434872, cached=9235892, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.84, five_min_avg=2.27, fifteen_min_avg=2.08, cpu_share=9.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:20:25.936 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:20:34.022 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12602856, partition_space_available_1k=68487628, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1496764, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12744, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51636, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6208988, free=335596, buffers=434624, cached=9196424, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.52, five_min_avg=2.2, fifteen_min_avg=2.07, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:21:25.947 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:21:34.021 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12611868, partition_space_available_1k=68478616, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1504688, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12796, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51628, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6220768, free=333912, buffers=434616, cached=9186336, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.87, five_min_avg=2.37, fifteen_min_avg=2.14, cpu_share=8.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:22:25.972 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:22:34.027 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12620068, partition_space_available_1k=68470416, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1523160, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12688, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51628, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6244000, free=327992, buffers=434608, cached=9169032, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.65, five_min_avg=2.38, fifteen_min_avg=2.15, cpu_share=8.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:23:25.951 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:23:35.035 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12651964, partition_space_available_1k=68438520, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1664132, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12800, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51576, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6325188, free=345368, buffers=434592, cached=9070484, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.5, five_min_avg=2.63, fifteen_min_avg=2.25, cpu_share=8.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:24:26.037 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:24:35.039 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12662724, partition_space_available_1k=68427760, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1660784, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12812, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51576, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6327292, free=333116, buffers=434596, cached=9080628, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.13, five_min_avg=2.38, fifteen_min_avg=2.18, cpu_share=7.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:25:25.957 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:25:31.450 7f00f761a048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 01/20/2025 02:25:34.038 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12669728, partition_space_available_1k=68420756, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1668628, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12916, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51576, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6333532, free=350532, buffers=434752, cached=9056816, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.58, five_min_avg=2.39, fifteen_min_avg=2.19, cpu_share=6.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:26:25.971 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 01/20/2025 02:26:34.038 7f00f9cfebc8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2650-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=12676224, partition_space_available_1k=68414260, percentage_partition_space_used=16, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1670872, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12960, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51576, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175632, used=6329584, free=337812, buffers=434788, cached=9073448, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.8, five_min_avg=2.18, fifteen_min_avg=2.13, cpu_share=7.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 01/20/2025 02:26:52.162 7f00f76a7f60 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map