12/26/2024 01:47:41.957 7f58ba74e088 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 12/26/2024 01:47:41.957 7f58ba74e088 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 12/26/2024 01:47:41.957 7f58ba74e088 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 12/26/2024 01:47:41.968 7f58b7fae438 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 12/26/2024 01:47:41.968 7f58b7fae438 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 12/26/2024 01:47:41.969 7f58b7fb5d68 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 12/26/2024 01:47:41.970 7f58b7fb5d68 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:47:41.970 7f58b7fb5d68 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 12/26/2024 01:47:41.971 7f58b7fb5d68 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 12/26/2024 01:47:41.971 7f58b7fb5d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:41.971 7f58b7fb5d68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-459-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/26/2024 01:47:41.971 7f58b7fb5d68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:41.971 7f58b7fb5d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:41.971 7f58b7fb5d68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-459-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1735177661970886 collector_name = collector_ip = collector_list = [ 10.0.0.29:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 12/26/2024 01:47:41.971 7f58b7fb5d68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:42.098 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 82759229265738995157763262963694656296957105393403020201063823485872643607873, 'start': '1735177660000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 96754105500470008687822100498534986006794306433864668054461391083094914235883, 'start': '1735177660000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 27443491011677369268406518851923585514320805869165962796868399676500446725452, 'start': '1735177659000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 12/26/2024 01:47:42.239 7f58bc329eb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:42.240 7f58bc329eb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-459-1 build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2024-12-26 01:04:16.666083", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7097-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 12/26/2024 01:47:42.240 7f58bc329eb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:42.241 7f58ba74e088 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-459-1', _node_type='Database', _instance_id='0', _timestamp=1735177662240872, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/26/2024 01:47:42.241 7f58ba74e088 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-459-1', _node_type='Database', _instance_id='0', _timestamp=1735177662241295, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1735177660000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1735177660000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1735177659000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2024-12-26 01:04:16.666083", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7097-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/26/2024 01:47:42.243 7f58b8052828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:42.243 7f58b8052828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-459-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/26/2024 01:47:42.243 7f58b8052828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:42.243 7f58b8052828 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:42.243 7f58b8052828 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-459-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1735177660000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1735177660000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1735177659000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2024-12-26 01:04:16.666083", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7097-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 12/26/2024 01:47:42.244 7f58b8052828 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:42.592 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 01:47:42.820 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 12/26/2024 01:47:42.822 7f58b810eeb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:42.822 7f58b810eeb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-459-1 database_usage = [ << disk_space_used_1k = 50515324 disk_space_available_1k = 30575160 analytics_db_size_1k = 4 >>, ] >> 12/26/2024 01:47:42.822 7f58b810eeb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:43.277 7f58ba74e088 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 12/26/2024 01:47:43.277 7f58ba74e088 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-459-1', _node_type='Database', _instance_id='0', _timestamp=1735177663277802, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/26/2024 01:47:43.279 7f58b7e718d0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:43.279 7f58b7e718d0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-459-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/26/2024 01:47:43.279 7f58b7e718d0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:44.385 7f58ba74e088 [contrail-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 2 12/26/2024 01:47:45.779 7f58ba74e088 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 12/26/2024 01:47:45.970 7f58ba773630 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/26/2024 01:47:45.972 7f58ba773630 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/26/2024 01:47:45.972 7f58ba773630 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/26/2024 01:47:45.972 7f58ba773630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/26/2024 01:47:45.972 7f58ba773630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-459-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.29:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/26/2024 01:47:45.972 7f58ba773630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/26/2024 01:47:45.972 7f58ba773630 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/26/2024 01:47:45.973 7f58ba773630 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/26/2024 01:47:45.973 7f58ba773630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:45.973 7f58ba773630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-459-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.29:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/26/2024 01:47:45.973 7f58ba773630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:45.973 7f58ba773630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:45.973 7f58ba773630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-459-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1735177661970886 collector_name = collector_ip = 10.0.0.29:8086 collector_list = [ 10.0.0.29:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/26/2024 01:47:45.973 7f58ba773630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/26/2024 01:47:49.974 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/26/2024 01:47:50.820 7f58ba74e088 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-459-1', _node_type='Database', _instance_id='0', _timestamp=1735177670820508, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1735177660000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1735177660000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1735177659000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2024-12-26 01:04:16.666083", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7097-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/26/2024 01:47:50.820 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50516712, partition_space_available_1k=30573772, percentage_partition_space_used=62, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1272416, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17280, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=49060, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=13191668, free=464332, buffers=688540, cached=18320576, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=7.61, five_min_avg=4.66, fifteen_min_avg=2.99, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 01:47:51.933 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 12/26/2024 01:47:51.933 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/26/2024 01:47:51.933 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/26/2024 01:47:51.934 7f58bb5a8c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-459-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.29:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/26/2024 01:47:51.934 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/26/2024 01:47:51.934 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/26/2024 01:47:51.934 7f58bb5a8c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-459-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1735177660000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1735177660000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1735177659000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2024-12-26 01:04:16.666083", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7097-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 12/26/2024 01:47:51.934 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/26/2024 01:47:51.934 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/26/2024 01:47:51.934 7f58bb5a8c18 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-459-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50516712 partition_space_available_1k = 30573772 percentage_partition_space_used = 62 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1272416 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 17280 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 49060 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 13191668 free = 464332 buffers = 688540 cached = 18320576 node_type = database-node >> system_cpu_usage = << one_min_avg = 7.61 five_min_avg = 4.66 fifteen_min_avg = 2.99 cpu_share = 0 node_type = database-node >> >> 12/26/2024 01:47:51.935 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/26/2024 01:47:51.936 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 12/26/2024 01:47:51.936 7f58bb5a8c18 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 12/26/2024 01:47:51.936 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 12/26/2024 01:47:51.937 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/26/2024 01:47:51.937 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/26/2024 01:47:51.950 7f58b808fa20 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001011 12/26/2024 01:47:51.951 7f58b808fa20 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 12/26/2024 01:47:51.951 7f58b815db70 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 12/26/2024 01:47:51.951 7f58b815db70 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 12/26/2024 01:47:51.953 7f58b815db70 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 12/26/2024 01:47:51.953 7f58b815db70 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:47:51.953 7f58b815db70 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 12/26/2024 01:47:51.953 7f58b815db70 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 12/26/2024 01:47:51.954 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 12/26/2024 01:47:51.955 7f58bb5a8c18 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 12/26/2024 01:48:06.579 7f58b88a6c18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:48:14.364 7f58b808fa20 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001432 12/26/2024 01:48:14.366 7f58b808fa20 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 12/26/2024 01:48:14.371 7f58b815eeb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 12/26/2024 01:48:14.371 7f58b815eeb8 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 3 12/26/2024 01:48:14.372 7f58b815eeb8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 12/26/2024 01:48:14.372 7f58b815eeb8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:48:14.372 7f58b815eeb8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 12/26/2024 01:48:14.372 7f58b7faed68 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 12/26/2024 01:48:14.373 7f58b7faed68 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 12/26/2024 01:48:22.923 7f58b7faed68 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:48:38.559 7f58ba7bcb70 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:48:42.557 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 01:48:43.884 7f58ba74e088 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-459-1', _node_type='Database', _instance_id='0', _timestamp=1735177723884501, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.29:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/26/2024 01:48:47.104 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-459-1', _node_type='Database', _instance_id='0', _timestamp=1735177727104570, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=1, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 12/26/2024 01:48:47.104 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 12/26/2024 01:48:52.143 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50873124, partition_space_available_1k=30217360, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1317524, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20268, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41876, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14678156, free=11795060, buffers=713640, cached=5478260, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=15.71, five_min_avg=7.73, fifteen_min_avg=4.16, cpu_share=9.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 01:48:58.853 7f58ba7af6d8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:49:06.779 7f58ba7af6d8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:49:12.389 7f58ba7af6d8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:49:42.562 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 01:49:44.493 7f58b8887a20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:49:46.376 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-459-1', _node_type='Database', _instance_id='0', _timestamp=1735177786376283, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 12/26/2024 01:49:51.409 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50928424, partition_space_available_1k=30162060, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1338296, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20548, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41796, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14806748, free=11513648, buffers=734840, cached=5609880, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=10.08, five_min_avg=7.64, fifteen_min_avg=4.36, cpu_share=5.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 01:49:51.949 7f58ba7a06d8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:50:07.998 7f58b810ec18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:50:13.070 7f58b810ec18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:50:17.442 7f58b7faed68 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:50:42.537 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 01:50:50.751 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50934696, partition_space_available_1k=30155788, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1357964, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20720, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41832, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14740968, free=11355332, buffers=736240, cached=5832576, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.5, five_min_avg=7.09, fifteen_min_avg=4.38, cpu_share=4.38, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 01:51:42.571 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 01:51:50.701 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50939648, partition_space_available_1k=30150836, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1340640, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20832, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41916, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14745188, free=11342980, buffers=736508, cached=5840440, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.87, five_min_avg=6.22, fifteen_min_avg=4.25, cpu_share=2.38, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 01:52:05.351 7f58b8900978 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:52:17.299 7f58b8124240 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:52:42.570 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 01:52:44.022 7f58b815e198 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:52:50.933 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50951900, partition_space_available_1k=30138584, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1377836, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20968, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41940, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14773732, free=10030920, buffers=774944, cached=7085520, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=7.08, five_min_avg=6.84, fifteen_min_avg=4.59, cpu_share=5.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 01:53:42.575 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 01:53:50.800 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=52276880, partition_space_available_1k=28813604, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1350236, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21068, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41944, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14809096, free=8757500, buffers=807928, cached=8290592, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.37, five_min_avg=6.6, fifteen_min_avg=4.65, cpu_share=3.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 01:54:28.653 7f58b88b04e0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:54:42.584 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 01:54:51.120 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=52161156, partition_space_available_1k=28929328, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1354220, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20784, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42208, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15800428, free=7689920, buffers=811060, cached=8363708, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.42, five_min_avg=5.94, fifteen_min_avg=4.54, cpu_share=4.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 01:55:03.203 7f58bb53c240 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:55:42.635 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 01:55:45.911 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-459-1', _node_type='Database', _instance_id='0', _timestamp=1735178145910925, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=78), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 12/26/2024 01:55:50.939 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=52188572, partition_space_available_1k=28901912, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1415432, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=25592, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42240, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=16298492, free=7128060, buffers=813932, cached=8424632, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.88, five_min_avg=5.94, fifteen_min_avg=4.62, cpu_share=5.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 01:56:42.624 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 01:56:51.205 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=52212236, partition_space_available_1k=28878248, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1411700, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=25588, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42244, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=16368668, free=7023076, buffers=813964, cached=8459408, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.83, five_min_avg=5.96, fifteen_min_avg=4.71, cpu_share=4.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 01:57:35.270 7f58b88c5438 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:57:42.888 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 01:57:48.223 7f58b8976438 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:57:55.070 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=52292924, partition_space_available_1k=28797560, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1413876, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=25744, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42268, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=16981928, free=6351400, buffers=814540, cached=8517248, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=10.63, five_min_avg=6.95, fifteen_min_avg=5.11, cpu_share=6.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 01:58:06.369 7f58bb5b0d68 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 01:58:42.637 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 01:58:51.203 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=52328120, partition_space_available_1k=28762364, percentage_partition_space_used=65, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1423304, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=25904, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42272, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=17579904, free=5730188, buffers=814548, cached=8540476, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=8.13, five_min_avg=6.91, fifteen_min_avg=5.21, cpu_share=4.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 01:59:42.624 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 01:59:50.830 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=52316492, partition_space_available_1k=28773992, percentage_partition_space_used=65, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1425620, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=26000, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42400, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=17186672, free=6097972, buffers=814552, cached=8565920, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.01, five_min_avg=6.28, fifteen_min_avg=5.1, cpu_share=4.38, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 01:59:51.646 7f58b8887828 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 02:00:42.652 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 02:00:51.241 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=52328800, partition_space_available_1k=28761684, percentage_partition_space_used=65, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1432604, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=25828, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42300, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=16688044, free=6582080, buffers=814568, cached=8580424, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.14, five_min_avg=5.82, fifteen_min_avg=5.02, cpu_share=4.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 02:01:42.640 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 02:01:51.734 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-459-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=52344292, partition_space_available_1k=28746192, percentage_partition_space_used=65, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1438532, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=25836, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42304, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=16473000, free=6778744, buffers=814584, cached=8598788, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.94, five_min_avg=5.71, fifteen_min_avg=5.03, cpu_share=4.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/26/2024 02:01:53.995 7f58ba5cea20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 02:01:57.337 7f58ba5cea20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 02:02:30.827 7f58ba7a3c18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 02:02:35.650 7f58ba7a3c18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/26/2024 02:02:42.670 7f58ba74e088 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/26/2024 02:02:46.143 7f58b814bd68 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map