03/27/2025 00:50:11.810 7f01ca0e6df0 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 03/27/2025 00:50:11.810 7f01ca0e6df0 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 03/27/2025 00:50:11.810 7f01ca0e6df0 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 03/27/2025 00:50:11.819 7f01c8377148 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 03/27/2025 00:50:11.819 7f01c8377148 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 03/27/2025 00:50:11.820 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 03/27/2025 00:50:11.821 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:50:11.821 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/27/2025 00:50:11.822 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 03/27/2025 00:50:11.822 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:11.822 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/27/2025 00:50:11.822 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:11.822 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:11.822 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1743036611821723 collector_name = collector_ip = collector_list = [ 10.0.0.27:8086, 10.0.0.43:8086, 10.0.0.36:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 03/27/2025 00:50:11.822 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:11.910 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 27033509557674735369252410832967981426389026023254792659707191156760120676380, 'start': '1743036609000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 101767161306006679411077654606483362286824757479624250507197460402162982892238, 'start': '1743036609000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 41976770631554692069153496412706645481288345010270615712649154738636373973829, 'start': '1743036609000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 03/27/2025 00:50:12.065 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:12.066 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-24 01:17:35.868579", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8260-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 4 num_core_per_socket = 4 num_thread_per_core = 1 >> running_package_version = nightly.r24.1.centos-nightly_r24.1_centos.el7 installed_package_version = nightly.r24.1.centos-nightly_r24.1_centos.el7 >> 03/27/2025 00:50:12.066 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:12.066 7f01ca0e6df0 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2962-2.', _node_type='Database', _instance_id='0', _timestamp=1743036612066740, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/27/2025 00:50:12.067 7f01ca0e6df0 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2962-2.', _node_type='Database', _instance_id='0', _timestamp=1743036612067044, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743036609000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743036609000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743036609000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-24 01:17:35.868579", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8260-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/27/2025 00:50:12.071 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:12.071 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/27/2025 00:50:12.071 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:12.071 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:12.071 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743036609000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743036609000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743036609000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-24 01:17:35.868579", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8260-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} >> 03/27/2025 00:50:12.071 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:12.442 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 00:50:12.784 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 03/27/2025 00:50:12.786 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:12.787 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. database_usage = [ << disk_space_used_1k = 6679728 disk_space_available_1k = 74410756 analytics_db_size_1k = 108 >>, ] >> 03/27/2025 00:50:12.787 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:13.495 7f01ca0e6df0 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 03/27/2025 00:50:13.495 7f01ca0e6df0 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2962-2.', _node_type='Database', _instance_id='0', _timestamp=1743036613495864, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/27/2025 00:50:13.499 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:13.500 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/27/2025 00:50:13.500 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:15.821 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/27/2025 00:50:15.823 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/27/2025 00:50:15.823 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/27/2025 00:50:15.823 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:15.824 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/27/2025 00:50:15.824 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:15.824 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/27/2025 00:50:15.825 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/27/2025 00:50:15.825 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:15.825 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/27/2025 00:50:15.825 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:15.825 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:15.825 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1743036611821723 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, 10.0.0.43:8086, 10.0.0.36:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/27/2025 00:50:15.825 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:16.637 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2962-2.', _node_type='Database', _instance_id='0', _timestamp=1743036616637423, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=1), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 03/27/2025 00:50:16.637 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 03/27/2025 00:50:16.639 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:16.639 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] CassandraStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. cassandra_compaction_task = << pending_compaction_tasks = 1 >> thread_pool_stats = [ << pool_name = ReadStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = CompactionExecutor active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MutationStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MemtableFlushWriter active = 0 pending = 0 all_time_blocked = 0 >>, ] >> 03/27/2025 00:50:16.639 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:17.796 7f01ca0e6df0 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 03/27/2025 00:50:19.823 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/27/2025 00:50:19.825 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/27/2025 00:50:19.826 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/27/2025 00:50:19.826 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:19.826 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.43:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/27/2025 00:50:19.826 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:19.826 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/27/2025 00:50:19.826 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/27/2025 00:50:19.827 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:19.827 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.43:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/27/2025 00:50:19.827 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:19.828 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:19.828 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1743036611821723 collector_name = collector_ip = 10.0.0.43:8086 collector_list = [ 10.0.0.27:8086, 10.0.0.43:8086, 10.0.0.36:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/27/2025 00:50:19.828 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:23.143 7f01ca0e6df0 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2962-2.', _node_type='Database', _instance_id='0', _timestamp=1743036623143619, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743036609000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743036609000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1743036609000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-24 01:17:35.868579", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8260-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/27/2025 00:50:23.144 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=6680696, partition_space_available_1k=74409788, percentage_partition_space_used=8, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1225960, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=7712, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51312, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=4279588, free=5524448, buffers=238752, cached=6132852, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.02, five_min_avg=1.07, fifteen_min_avg=0.56, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 00:50:23.144 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:23.144 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743036609000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743036609000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1743036609000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly.r24.1.centos", "build-time" : "2025-03-24 01:17:35.868579", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-8260-1.novalocal", "build-id" : "nightly.r24.1.centos", "build-number" : "nightly_r24.1_centos.el7"}]} >> 03/27/2025 00:50:23.144 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:23.144 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:23.144 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 6680696 partition_space_available_1k = 74409788 percentage_partition_space_used = 8 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1225960 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 7712 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 51312 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 16175640 used = 4279588 free = 5524448 buffers = 238752 cached = 6132852 node_type = database-node >> system_cpu_usage = << one_min_avg = 1.02 five_min_avg = 1.07 fifteen_min_avg = 0.56 cpu_share = 0 node_type = database-node >> >> 03/27/2025 00:50:23.145 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:23.826 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/27/2025 00:50:23.828 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/27/2025 00:50:23.828 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/27/2025 00:50:23.828 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:23.828 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.36:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/27/2025 00:50:23.829 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:23.829 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/27/2025 00:50:23.829 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/27/2025 00:50:23.829 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:23.829 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.36:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/27/2025 00:50:23.829 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:23.829 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:23.830 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1743036611821723 collector_name = collector_ip = 10.0.0.36:8086 collector_list = [ 10.0.0.27:8086, 10.0.0.43:8086, 10.0.0.36:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/27/2025 00:50:23.830 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:27.829 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/27/2025 00:50:27.830 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/27/2025 00:50:27.831 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/27/2025 00:50:27.831 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:27.831 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/27/2025 00:50:27.831 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:27.832 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/27/2025 00:50:27.832 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/27/2025 00:50:27.832 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:27.832 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/27/2025 00:50:27.832 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:27.832 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:27.832 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1743036611821723 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, 10.0.0.43:8086, 10.0.0.36:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/27/2025 00:50:27.832 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:31.831 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/27/2025 00:50:31.833 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/27/2025 00:50:31.833 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/27/2025 00:50:31.834 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:31.834 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.43:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/27/2025 00:50:31.834 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:31.834 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/27/2025 00:50:31.834 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/27/2025 00:50:31.834 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:31.835 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.43:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/27/2025 00:50:31.835 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:31.835 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:31.835 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1743036611821723 collector_name = collector_ip = 10.0.0.43:8086 collector_list = [ 10.0.0.27:8086, 10.0.0.43:8086, 10.0.0.36:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/27/2025 00:50:31.835 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:35.836 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/27/2025 00:50:35.838 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/27/2025 00:50:35.839 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/27/2025 00:50:35.839 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:35.839 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.36:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/27/2025 00:50:35.839 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:35.839 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/27/2025 00:50:35.840 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/27/2025 00:50:35.840 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:35.840 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.36:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/27/2025 00:50:35.840 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:35.840 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:35.841 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1743036611821723 collector_name = collector_ip = 10.0.0.36:8086 collector_list = [ 10.0.0.27:8086, 10.0.0.43:8086, 10.0.0.36:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/27/2025 00:50:35.841 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:39.840 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/27/2025 00:50:39.841 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/27/2025 00:50:39.842 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/27/2025 00:50:39.842 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:39.842 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/27/2025 00:50:39.842 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:39.842 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/27/2025 00:50:39.843 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/27/2025 00:50:39.843 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:39.843 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/27/2025 00:50:39.843 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:39.843 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:39.843 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1743036611821723 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, 10.0.0.43:8086, 10.0.0.36:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/27/2025 00:50:39.843 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:43.842 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/27/2025 00:50:43.844 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/27/2025 00:50:43.845 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/27/2025 00:50:43.845 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:43.845 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.43:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/27/2025 00:50:43.845 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:43.845 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/27/2025 00:50:43.846 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/27/2025 00:50:43.846 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:43.846 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.43:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/27/2025 00:50:43.846 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:43.846 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:43.846 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1743036611821723 collector_name = collector_ip = 10.0.0.43:8086 collector_list = [ 10.0.0.27:8086, 10.0.0.43:8086, 10.0.0.36:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/27/2025 00:50:43.847 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:47.848 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/27/2025 00:50:47.850 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/27/2025 00:50:47.850 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/27/2025 00:50:47.850 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:47.851 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.36:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/27/2025 00:50:47.851 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:47.851 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/27/2025 00:50:47.851 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/27/2025 00:50:47.851 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:47.851 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.36:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/27/2025 00:50:47.852 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:47.852 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:47.852 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1743036611821723 collector_name = collector_ip = 10.0.0.36:8086 collector_list = [ 10.0.0.27:8086, 10.0.0.43:8086, 10.0.0.36:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/27/2025 00:50:47.852 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:51.850 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/27/2025 00:50:51.852 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/27/2025 00:50:51.852 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/27/2025 00:50:51.852 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:51.852 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/27/2025 00:50:51.852 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:51.852 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/27/2025 00:50:51.853 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/27/2025 00:50:51.853 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:51.853 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/27/2025 00:50:51.853 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:51.853 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:51.853 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1743036611821723 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, 10.0.0.43:8086, 10.0.0.36:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/27/2025 00:50:51.854 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:55.856 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/27/2025 00:50:55.858 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/27/2025 00:50:55.858 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/27/2025 00:50:55.858 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:55.858 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.43:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/27/2025 00:50:55.858 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:55.858 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/27/2025 00:50:55.859 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/27/2025 00:50:55.859 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:55.859 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.43:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/27/2025 00:50:55.859 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:55.859 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:55.859 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1743036611821723 collector_name = collector_ip = 10.0.0.43:8086 collector_list = [ 10.0.0.27:8086, 10.0.0.43:8086, 10.0.0.36:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/27/2025 00:50:55.860 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:59.860 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/27/2025 00:50:59.862 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/27/2025 00:50:59.862 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/27/2025 00:50:59.862 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:59.862 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.36:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/27/2025 00:50:59.863 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:50:59.863 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/27/2025 00:50:59.863 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/27/2025 00:50:59.863 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:59.863 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.36:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/27/2025 00:50:59.863 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:59.863 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:50:59.863 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1743036611821723 collector_name = collector_ip = 10.0.0.36:8086 collector_list = [ 10.0.0.27:8086, 10.0.0.43:8086, 10.0.0.36:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/27/2025 00:50:59.864 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:51:03.865 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/27/2025 00:51:03.867 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 03/27/2025 00:51:03.867 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/27/2025 00:51:03.867 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:51:03.867 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/27/2025 00:51:03.867 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:51:03.867 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/27/2025 00:51:03.868 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/27/2025 00:51:03.868 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:51:03.868 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/27/2025 00:51:03.868 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:51:03.868 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:51:03.868 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1743036611821723 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, 10.0.0.43:8086, 10.0.0.36:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/27/2025 00:51:03.868 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/27/2025 00:51:07.867 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/27/2025 00:51:07.869 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 03/27/2025 00:51:07.869 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/27/2025 00:51:07.869 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:51:07.869 7f01c48e0648 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2962-2. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.43:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/27/2025 00:51:07.870 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/27/2025 00:51:07.870 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/27/2025 00:51:07.870 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 03/27/2025 00:51:07.870 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/27/2025 00:51:07.870 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/27/2025 00:51:07.870 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/27/2025 00:51:12.326 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 00:51:12.685 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/27/2025 00:51:13.344 7f01ca0e6df0 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2962-2.', _node_type='Database', _instance_id='0', _timestamp=1743036673344206, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.43:8086'], status='Initializing', description='Connect to ClientInit on EvTcpConnected')]), flag_infos=None, description='Collector:Collector[Connect to ClientInit on EvTcpConnected] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 03/27/2025 00:51:13.346 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/27/2025 00:51:16.438 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2962-2.', _node_type='Database', _instance_id='0', _timestamp=1743036676438535, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=1, pending=1, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 03/27/2025 00:51:16.441 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/27/2025 00:51:19.162 7f01c4769148 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001016 03/27/2025 00:51:19.163 7f01c4769148 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/27/2025 00:51:19.163 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 03/27/2025 00:51:19.163 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 03/27/2025 00:51:19.164 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 03/27/2025 00:51:19.164 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 03/27/2025 00:51:19.164 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:51:19.164 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/27/2025 00:51:19.164 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 03/27/2025 00:51:19.165 7f01c48e0e48 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/27/2025 00:51:19.166 7f01c48e0e48 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/27/2025 00:51:22.158 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=6941044, partition_space_available_1k=74149440, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1305232, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=9600, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51784, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=4554152, free=4907156, buffers=251036, cached=6463296, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.44, five_min_avg=1.12, fifteen_min_avg=0.6, cpu_share=5.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 00:51:33.715 7f01c4769148 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001582 03/27/2025 00:51:33.715 7f01c4769148 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/27/2025 00:51:33.716 7f01c48e0648 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 03/27/2025 00:51:33.716 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 4 03/27/2025 00:51:33.716 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 03/27/2025 00:51:33.716 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 03/27/2025 00:51:33.717 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:51:33.717 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/27/2025 00:51:33.717 7f01c48e0e48 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/27/2025 00:51:33.718 7f01c48e0e48 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/27/2025 00:52:12.363 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 00:52:16.551 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-2962-2.', _node_type='Database', _instance_id='0', _timestamp=1743036736551256, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=2), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 03/27/2025 00:52:22.176 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7125352, partition_space_available_1k=73965132, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1337568, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10064, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51852, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5501352, free=3783104, buffers=261384, cached=6629800, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.61, five_min_avg=1.51, fifteen_min_avg=0.77, cpu_share=13.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 00:53:12.361 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 00:53:22.172 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7187840, partition_space_available_1k=73902644, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1338424, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10112, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51864, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5518696, free=3700064, buffers=262672, cached=6694208, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.15, five_min_avg=1.86, fifteen_min_avg=0.94, cpu_share=12.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 00:54:06.000 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:54:12.422 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 00:54:22.195 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7661744, partition_space_available_1k=73428740, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1376672, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10216, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51896, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5591624, free=3135856, buffers=269736, cached=7178424, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.99, five_min_avg=2.06, fifteen_min_avg=1.06, cpu_share=14.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 00:54:53.776 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:55:12.416 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 00:55:14.089 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:55:23.197 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7667628, partition_space_available_1k=73422856, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1350164, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10356, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52308, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5623932, free=3084364, buffers=271720, cached=7195624, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.64, five_min_avg=2.42, fifteen_min_avg=1.26, cpu_share=13.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 00:55:34.743 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:55:54.398 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:56:12.395 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 00:56:15.584 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:56:23.200 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7670604, partition_space_available_1k=73419880, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1357016, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10504, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52404, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5729592, free=2972256, buffers=273352, cached=7200440, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.63, five_min_avg=2.67, fifteen_min_avg=1.42, cpu_share=13.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 00:56:35.832 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:56:55.390 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:57:12.431 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 00:57:15.993 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:57:23.204 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7675712, partition_space_available_1k=73414772, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1383764, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10612, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52400, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5779400, free=2921416, buffers=274544, cached=7200280, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.31, five_min_avg=2.77, fifteen_min_avg=1.54, cpu_share=13.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 00:57:36.232 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:57:56.056 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:58:12.419 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 00:58:17.288 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:58:23.221 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7688820, partition_space_available_1k=73401664, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1367240, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10532, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52404, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5825640, free=2863428, buffers=275496, cached=7211076, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.82, five_min_avg=3.03, fifteen_min_avg=1.7, cpu_share=13.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 00:58:38.583 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:58:58.472 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:59:12.456 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 00:59:19.394 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:59:22.202 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7698584, partition_space_available_1k=73391900, percentage_partition_space_used=9, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1370880, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10572, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52408, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5869364, free=2804832, buffers=276348, cached=7225096, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.93, five_min_avg=3.22, fifteen_min_avg=1.85, cpu_share=13.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 00:59:40.081 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 00:59:59.900 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:00:12.540 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:00:12.663 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:00:22.201 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7704608, partition_space_available_1k=73385876, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1377772, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10604, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52412, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5914264, free=2752016, buffers=276844, cached=7232516, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.03, five_min_avg=3.08, fifteen_min_avg=1.9, cpu_share=12.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:00:25.749 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:00:38.350 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:00:50.964 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:01:12.029 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:01:12.561 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:01:22.214 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7710936, partition_space_available_1k=73379548, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1383144, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10648, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52444, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5950856, free=2706644, buffers=277120, cached=7241020, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.65, five_min_avg=3.26, fifteen_min_avg=2.03, cpu_share=13.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:01:33.735 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:01:54.058 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:02:12.495 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:02:14.651 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:02:22.229 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7719544, partition_space_available_1k=73370940, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1388492, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10664, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52460, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=5993016, free=2651568, buffers=277392, cached=7253664, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.52, five_min_avg=3.31, fifteen_min_avg=2.13, cpu_share=12.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:02:35.561 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:02:55.324 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:03:12.501 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:03:16.057 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:03:23.214 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7736400, partition_space_available_1k=73354084, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1393296, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10792, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52464, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6021268, free=2619460, buffers=277628, cached=7257284, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.6, five_min_avg=3.63, fifteen_min_avg=2.31, cpu_share=12.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:03:36.920 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:03:57.112 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:04:12.457 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:04:17.803 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:04:22.228 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7739304, partition_space_available_1k=73351180, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1400788, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10628, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52492, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6081744, free=2541784, buffers=277808, cached=7274304, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.27, five_min_avg=3.71, fifteen_min_avg=2.42, cpu_share=13.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:04:38.293 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:04:58.532 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:05:12.447 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:05:19.118 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:05:22.217 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7746392, partition_space_available_1k=73344092, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1412800, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10676, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52496, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6121688, free=2492936, buffers=277892, cached=7283124, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.12, five_min_avg=3.49, fifteen_min_avg=2.43, cpu_share=12.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:05:40.246 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:06:01.341 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:06:12.517 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:06:21.624 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:06:22.223 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7760044, partition_space_available_1k=73330440, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1409504, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10712, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52500, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6137884, free=2470364, buffers=277928, cached=7289464, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.91, five_min_avg=3.6, fifteen_min_avg=2.53, cpu_share=13.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:06:41.993 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:07:02.901 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:07:12.465 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:07:22.232 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7767560, partition_space_available_1k=73322924, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1416592, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10724, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52504, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6166344, free=2429860, buffers=277972, cached=7301464, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.99, five_min_avg=3.63, fifteen_min_avg=2.61, cpu_share=12.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:07:23.847 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:07:43.963 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:08:04.588 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:08:12.500 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:08:22.236 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7774208, partition_space_available_1k=73316276, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1419804, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10808, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52508, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6149748, free=2444496, buffers=278012, cached=7303384, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.92, five_min_avg=3.65, fifteen_min_avg=2.68, cpu_share=13.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:08:25.480 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:08:45.821 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:09:06.418 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:09:12.472 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:09:22.235 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7781540, partition_space_available_1k=73308944, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1427188, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10808, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52512, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6161604, free=2424024, buffers=278032, cached=7311980, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.9, five_min_avg=3.68, fifteen_min_avg=2.75, cpu_share=13.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:09:27.141 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:09:47.265 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:10:07.996 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:10:12.521 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:10:22.237 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7787568, partition_space_available_1k=73302916, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1433200, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10788, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52516, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6165956, free=2408960, buffers=278056, cached=7322668, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.09, five_min_avg=3.46, fifteen_min_avg=2.73, cpu_share=12.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:10:28.902 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:10:49.168 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:11:10.381 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:11:12.547 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:11:22.251 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7801756, partition_space_available_1k=73288728, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1438500, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10848, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52520, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6165752, free=2400800, buffers=278080, cached=7331008, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.13, five_min_avg=3.43, fifteen_min_avg=2.77, cpu_share=12.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:11:30.749 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:11:51.138 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:12:12.032 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:12:12.518 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:12:22.269 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7786208, partition_space_available_1k=73304276, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1446044, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10860, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52524, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6190376, free=2387308, buffers=278092, cached=7319864, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.06, five_min_avg=3.34, fifteen_min_avg=2.78, cpu_share=12.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:12:32.604 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:12:53.072 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:13:12.524 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:13:14.041 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:13:22.248 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7807216, partition_space_available_1k=73283268, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1447536, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10872, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52528, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6213056, free=2361680, buffers=278104, cached=7322800, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.51, five_min_avg=3.15, fifteen_min_avg=2.75, cpu_share=12.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:13:34.806 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:13:54.702 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:14:12.507 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:14:15.051 7f01c48e0648 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:14:22.266 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7807356, partition_space_available_1k=73283128, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1449972, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10956, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52532, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6209876, free=2353952, buffers=278116, cached=7333696, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.85, five_min_avg=3.45, fifteen_min_avg=2.88, cpu_share=12.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:14:35.562 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:14:45.629 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:14:54.962 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 03/27/2025 01:15:12.471 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 03/27/2025 01:15:22.269 7f01ca0e6df0 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-2962-2.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=7821784, partition_space_available_1k=73268700, percentage_partition_space_used=10, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1456424, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10964, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=52544, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=16175640, used=6222060, free=2342188, buffers=278124, cached=7333268, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.51, five_min_avg=3.41, fifteen_min_avg=2.91, cpu_share=12.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 03/27/2025 01:15:38.837 7f01c48e0e48 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map