12/23/2024 01:48:22.932 7f2b4622b3c8 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 12/23/2024 01:48:22.932 7f2b4622b3c8 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 12/23/2024 01:48:22.932 7f2b4622b3c8 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 12/23/2024 01:48:22.943 7f2b438f1390 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 12/23/2024 01:48:22.943 7f2b438f1390 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 12/23/2024 01:48:22.944 7f2b438f3cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 12/23/2024 01:48:22.945 7f2b438f3cc0 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:48:22.945 7f2b438f3cc0 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 12/23/2024 01:48:22.946 7f2b438f3cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 12/23/2024 01:48:22.946 7f2b438f3cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:22.946 7f2b438f3cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/23/2024 01:48:22.946 7f2b438f3cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:22.946 7f2b438f3cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:22.946 7f2b438f3cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-458-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1734918502945653 collector_name = collector_ip = collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 12/23/2024 01:48:22.946 7f2b438f3cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:23.002 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 81399002839450730545816597870067703991055275062828344811868813272036645782707, 'start': '1734918500000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 43091166700964187674721386831075158736971257317861879028208263831238286338849, 'start': '1734918500000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 26136486853276277101764018138192755511705851880324399662301471424946569996626, 'start': '1734918500000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 12/23/2024 01:48:23.214 7f2b47e1ceb8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:23.215 7f2b47e1ceb8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2024-12-23 01:03:26.567380", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7088-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 12/23/2024 01:48:23.215 7f2b47e1ceb8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:23.216 7f2b4622b3c8 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-458-1', _node_type='Database', _instance_id='0', _timestamp=1734918503216180, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/23/2024 01:48:23.216 7f2b4622b3c8 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-458-1', _node_type='Database', _instance_id='0', _timestamp=1734918503216604, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1734918500000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1734918500000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1734918500000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2024-12-23 01:03:26.567380", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7088-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/23/2024 01:48:23.219 7f2b4d3220f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:23.220 7f2b4d3220f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/23/2024 01:48:23.220 7f2b4d3220f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:23.220 7f2b4d3220f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:23.220 7f2b4d3220f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1734918500000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1734918500000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1734918500000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2024-12-23 01:03:26.567380", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7088-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 12/23/2024 01:48:23.220 7f2b4d3220f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:23.502 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 01:48:23.642 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 12/23/2024 01:48:23.644 7f2b43a67e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:23.645 7f2b43a67e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 database_usage = [ << disk_space_used_1k = 50442184 disk_space_available_1k = 30648300 analytics_db_size_1k = 4 >>, ] >> 12/23/2024 01:48:23.645 7f2b43a67e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:24.201 7f2b4622b3c8 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 12/23/2024 01:48:24.201 7f2b4622b3c8 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-458-1', _node_type='Database', _instance_id='0', _timestamp=1734918504201711, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/23/2024 01:48:24.203 7f2b471266d8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:24.203 7f2b471266d8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/23/2024 01:48:24.204 7f2b471266d8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:25.440 7f2b4622b3c8 [contrail-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 2 12/23/2024 01:48:26.930 7f2b4622b3c8 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 12/23/2024 01:48:26.945 7f2b438842e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/23/2024 01:48:26.947 7f2b438842e8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/23/2024 01:48:26.948 7f2b438842e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/23/2024 01:48:26.948 7f2b438842e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:48:26.948 7f2b438842e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/23/2024 01:48:26.948 7f2b438842e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:48:26.948 7f2b438842e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/23/2024 01:48:26.949 7f2b438842e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/23/2024 01:48:26.949 7f2b438842e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:26.949 7f2b438842e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/23/2024 01:48:26.949 7f2b438842e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:26.949 7f2b438842e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:26.950 7f2b438842e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-458-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1734918502945653 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/23/2024 01:48:26.950 7f2b438842e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:30.949 7f2b4389f630 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/23/2024 01:48:30.951 7f2b4389f630 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/23/2024 01:48:30.951 7f2b4389f630 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/23/2024 01:48:30.951 7f2b4389f630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:48:30.951 7f2b4389f630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/23/2024 01:48:30.951 7f2b4389f630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:48:30.952 7f2b4389f630 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/23/2024 01:48:30.952 7f2b4389f630 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/23/2024 01:48:30.952 7f2b4389f630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:30.953 7f2b4389f630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/23/2024 01:48:30.953 7f2b4389f630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:30.953 7f2b4389f630 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:30.953 7f2b4389f630 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-458-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1734918502945653 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/23/2024 01:48:30.953 7f2b4389f630 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:31.969 7f2b4622b3c8 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-458-1', _node_type='Database', _instance_id='0', _timestamp=1734918511969327, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1734918500000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1734918500000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1734918500000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2024-12-23 01:03:26.567380", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7088-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/23/2024 01:48:31.969 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50443580, partition_space_available_1k=30646904, percentage_partition_space_used=62, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1292208, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=20492, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=69532, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=10784420, free=616148, buffers=887700, cached=20376848, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.88, five_min_avg=3.24, fifteen_min_avg=2.53, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 01:48:31.970 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:31.970 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1734918500000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1734918500000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1734918500000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2024-12-23 01:03:26.567380", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-7088-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 12/23/2024 01:48:31.970 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:31.970 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:31.970 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50443580 partition_space_available_1k = 30646904 percentage_partition_space_used = 62 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1292208 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 20492 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 69532 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 10784420 free = 616148 buffers = 887700 cached = 20376848 node_type = database-node >> system_cpu_usage = << one_min_avg = 3.88 five_min_avg = 3.24 fifteen_min_avg = 2.53 cpu_share = 0 node_type = database-node >> >> 12/23/2024 01:48:31.970 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:34.955 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/23/2024 01:48:39.962 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/23/2024 01:48:39.962 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/23/2024 01:48:39.963 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:48:39.963 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/23/2024 01:48:39.963 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:48:39.963 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/23/2024 01:48:39.964 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/23/2024 01:48:39.964 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:39.964 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/23/2024 01:48:39.964 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:39.964 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:39.964 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-458-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1734918502945653 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/23/2024 01:48:39.964 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:43.966 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/23/2024 01:48:44.693 7f2b43875978 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:48:48.971 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/23/2024 01:48:48.971 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/23/2024 01:48:48.971 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:48:48.971 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/23/2024 01:48:48.971 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:48:48.971 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/23/2024 01:48:48.972 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/23/2024 01:48:48.972 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:48.972 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/23/2024 01:48:48.972 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:48.972 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:48.973 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-458-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1734918502945653 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/23/2024 01:48:48.973 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:52.975 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/23/2024 01:48:56.752 7f2b4387b8d0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:48:57.978 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/23/2024 01:48:57.978 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/23/2024 01:48:57.979 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:48:57.979 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/23/2024 01:48:57.979 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:48:57.979 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/23/2024 01:48:57.980 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/23/2024 01:48:57.980 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:57.980 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/23/2024 01:48:57.980 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:57.980 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:48:57.980 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-458-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1734918502945653 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/23/2024 01:48:57.980 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:01.982 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/23/2024 01:49:05.552 7f2b4d322ac8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:49:06.986 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/23/2024 01:49:06.986 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/23/2024 01:49:06.986 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:06.986 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/23/2024 01:49:06.986 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:06.986 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/23/2024 01:49:06.987 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/23/2024 01:49:06.987 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:06.987 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/23/2024 01:49:06.987 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:06.987 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:06.987 7f2b45de1cc0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-458-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1734918502945653 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/23/2024 01:49:06.987 7f2b45de1cc0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:10.987 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/23/2024 01:49:16.030 7f2b43909e10 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/23/2024 01:49:16.031 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/23/2024 01:49:16.031 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:16.031 7f2b43909e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/23/2024 01:49:16.031 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:16.031 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/23/2024 01:49:16.032 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/23/2024 01:49:16.032 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:16.032 7f2b43909e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/23/2024 01:49:16.032 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:16.032 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:16.033 7f2b43909e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-458-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1734918502945653 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/23/2024 01:49:16.033 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:20.035 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/23/2024 01:49:23.585 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 01:49:25.046 7f2b43909e10 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/23/2024 01:49:25.046 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/23/2024 01:49:25.047 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:25.047 7f2b43909e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/23/2024 01:49:25.047 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:25.047 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:25.047 7f2b43909e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 database_usage = [ << disk_space_used_1k = 50886100 disk_space_available_1k = 30204384 analytics_db_size_1k = 988 >>, ] >> 12/23/2024 01:49:25.048 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:25.048 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/23/2024 01:49:25.048 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/23/2024 01:49:25.049 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:25.049 7f2b43909e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/23/2024 01:49:25.049 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:25.049 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:25.049 7f2b43909e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-458-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1734918502945653 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 12/23/2024 01:49:25.049 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:26.003 7f2b4622b3c8 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-458-1', _node_type='Database', _instance_id='0', _timestamp=1734918566003096, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.12:8086'], status='Down', description='Connect to Idle on EvTcpConnectFail')]), flag_infos=None, description='Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 12/23/2024 01:49:26.005 7f2b46246048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:26.005 7f2b46246048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/23/2024 01:49:26.005 7f2b46246048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:29.049 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/23/2024 01:49:34.065 7f2b43909e10 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/23/2024 01:49:34.065 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/23/2024 01:49:34.065 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:34.065 7f2b43909e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/23/2024 01:49:34.065 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:34.065 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/23/2024 01:49:34.066 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/23/2024 01:49:34.066 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:34.066 7f2b43909e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/23/2024 01:49:34.066 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:34.066 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:34.066 7f2b43909e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-458-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1734918502945653 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 12/23/2024 01:49:34.067 7f2b43909e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:36.222 7f2b4622b3c8 [contrail-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: timed out 12/23/2024 01:49:38.065 7f2b438906d8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/23/2024 01:49:38.863 7f2b438daac8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:49:41.937 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50909036, partition_space_available_1k=30181448, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1320052, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22280, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=69848, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=13629824, free=690992, buffers=686428, cached=17657872, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=14.67, five_min_avg=5.82, fifteen_min_avg=3.44, cpu_share=6.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 01:49:43.074 7f2b438906d8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 12/23/2024 01:49:43.075 7f2b438906d8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/23/2024 01:49:43.075 7f2b438906d8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:43.075 7f2b438906d8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/23/2024 01:49:43.076 7f2b438906d8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:43.076 7f2b438906d8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:43.077 7f2b438906d8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50909036 partition_space_available_1k = 30181448 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1320052 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 22280 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 69848 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 13629824 free = 690992 buffers = 686428 cached = 17657872 node_type = database-node >> system_cpu_usage = << one_min_avg = 14.67 five_min_avg = 5.82 fifteen_min_avg = 3.44 cpu_share = 6.0 node_type = database-node >> >> 12/23/2024 01:49:43.077 7f2b438906d8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:43.077 7f2b438906d8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/23/2024 01:49:43.078 7f2b438906d8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/23/2024 01:49:43.078 7f2b438906d8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:43.078 7f2b438906d8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/23/2024 01:49:43.079 7f2b438906d8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:43.079 7f2b438906d8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:43.079 7f2b438906d8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-458-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1734918502945653 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 12/23/2024 01:49:43.080 7f2b438906d8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:47.078 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/23/2024 01:49:51.322 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 12/23/2024 01:49:51.322 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/23/2024 01:49:51.322 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:51.322 7f2b46289e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/23/2024 01:49:51.323 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:51.323 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 12/23/2024 01:49:51.323 7f2b46289e10 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 12/23/2024 01:49:51.324 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 12/23/2024 01:49:51.324 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/23/2024 01:49:51.324 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/23/2024 01:49:51.348 7f2b4d322ac8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 12/23/2024 01:49:51.348 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 12/23/2024 01:49:51.349 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 12/23/2024 01:49:51.349 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:51.349 7f2b46289e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 12/23/2024 01:49:51.349 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:51.349 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:51.350 7f2b46289e10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-458-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1734918502945653 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 12/23/2024 01:49:51.350 7f2b46289e10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:55.358 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/23/2024 01:49:55.393 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 12/23/2024 01:49:55.394 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/23/2024 01:49:55.395 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:55.395 7f2b462424e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/23/2024 01:49:55.395 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:55.395 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 12/23/2024 01:49:55.395 7f2b462424e0 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 12/23/2024 01:49:55.396 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 12/23/2024 01:49:55.396 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/23/2024 01:49:55.396 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/23/2024 01:49:55.415 7f2b4626eeb8 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 12/23/2024 01:49:55.416 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 12/23/2024 01:49:55.416 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 12/23/2024 01:49:55.418 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:55.419 7f2b462424e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 12/23/2024 01:49:55.419 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:55.419 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:55.419 7f2b462424e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-458-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 2 pid = 1 http_port = 8103 start_time = 1734918502945653 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 12/23/2024 01:49:55.419 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/23/2024 01:49:59.418 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/23/2024 01:49:59.443 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 12/23/2024 01:49:59.444 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/23/2024 01:49:59.444 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:59.444 7f2b462424e0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-458-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/23/2024 01:49:59.444 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/23/2024 01:49:59.444 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 12/23/2024 01:49:59.444 7f2b462424e0 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 12/23/2024 01:49:59.445 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 12/23/2024 01:49:59.445 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/23/2024 01:49:59.446 7f2b462424e0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/23/2024 01:49:59.457 7f2b442f68d0 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001012 12/23/2024 01:49:59.458 7f2b442f68d0 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 12/23/2024 01:49:59.459 7f2b48947438 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 12/23/2024 01:49:59.459 7f2b48947438 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 12/23/2024 01:49:59.459 7f2b48947438 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 12/23/2024 01:49:59.459 7f2b48947438 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:49:59.459 7f2b48947438 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 12/23/2024 01:49:59.459 7f2b48947438 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 12/23/2024 01:49:59.460 7f2b46265ac8 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 12/23/2024 01:49:59.461 7f2b46265ac8 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 12/23/2024 01:50:01.208 7f2b442f68d0 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001433 12/23/2024 01:50:01.208 7f2b442f68d0 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 12/23/2024 01:50:01.209 7f2b4630a048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 12/23/2024 01:50:01.209 7f2b4630a048 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 3 12/23/2024 01:50:01.209 7f2b4630a048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 12/23/2024 01:50:01.210 7f2b4630a048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:50:01.210 7f2b4630a048 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 12/23/2024 01:50:01.210 7f2b439c5828 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 12/23/2024 01:50:01.211 7f2b439c5828 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 12/23/2024 01:50:09.287 7f2b439c5828 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:50:16.614 7f2b462a6b70 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:50:23.605 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 01:50:24.542 7f2b428cae10 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:50:28.420 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-458-1', _node_type='Database', _instance_id='0', _timestamp=1734918628419998, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 12/23/2024 01:50:28.420 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 12/23/2024 01:50:33.706 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50857520, partition_space_available_1k=30232964, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1333936, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22740, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53476, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14433400, free=12089080, buffers=706060, cached=5436576, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=12.91, five_min_avg=7.04, fifteen_min_avg=4.01, cpu_share=7.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 01:50:34.304 7f2b4607d828 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:51:23.562 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 01:51:33.712 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50920884, partition_space_available_1k=30169600, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1333276, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22876, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53508, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14637508, free=11725200, buffers=719748, cached=5582660, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=11.48, five_min_avg=7.67, fifteen_min_avg=4.41, cpu_share=7.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 01:52:23.553 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 01:52:32.724 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50928824, partition_space_available_1k=30161660, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1339944, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22968, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53516, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14626724, free=11716964, buffers=719832, cached=5601596, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=7.7, five_min_avg=7.35, fifteen_min_avg=4.51, cpu_share=3.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 01:53:23.565 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 01:53:32.729 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50933512, partition_space_available_1k=30156972, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1344176, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23104, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53860, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14641976, free=11676484, buffers=724536, cached=5622120, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.2, five_min_avg=6.37, fifteen_min_avg=4.35, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 01:54:11.765 7f2b463fbcc0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:54:17.671 7f2b43aae240 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:54:23.600 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 01:54:32.746 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50945064, partition_space_available_1k=30145420, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1379868, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23268, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53876, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14802052, free=11382860, buffers=725980, cached=5754224, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.37, five_min_avg=6.23, fifteen_min_avg=4.42, cpu_share=4.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 01:54:37.483 7f2b47d8f438 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:54:43.540 7f2b47d8f438 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:55:24.605 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 01:55:37.797 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50950696, partition_space_available_1k=30139788, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1353064, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23416, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53876, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14779680, free=9953912, buffers=787320, cached=7144204, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=9.31, five_min_avg=6.91, fifteen_min_avg=4.76, cpu_share=5.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 01:56:23.589 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 01:56:32.753 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50957888, partition_space_available_1k=30132596, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1355780, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22976, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53880, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14715480, free=10011400, buffers=787320, cached=7150916, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.05, five_min_avg=6.13, fifteen_min_avg=4.61, cpu_share=3.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 01:57:23.588 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 01:57:27.258 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-458-1', _node_type='Database', _instance_id='0', _timestamp=1734919047258026, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=2, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 12/23/2024 01:57:32.753 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50963436, partition_space_available_1k=30127048, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1380488, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23064, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53888, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14776164, free=9943288, buffers=787324, cached=7158340, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.24, five_min_avg=5.42, fifteen_min_avg=4.46, cpu_share=2.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 01:58:23.594 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 01:58:26.989 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-458-1', _node_type='Database', _instance_id='0', _timestamp=1734919106989882, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 12/23/2024 01:58:32.760 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50968112, partition_space_available_1k=30122372, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1365908, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23128, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53896, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14761916, free=9951056, buffers=787328, cached=7164816, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.24, five_min_avg=4.75, fifteen_min_avg=4.29, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 01:59:23.630 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 01:59:28.069 7f2b462a6390 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:59:32.774 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50977628, partition_space_available_1k=30112856, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1367476, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23260, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53908, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14857952, free=9844612, buffers=787424, cached=7175128, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.21, five_min_avg=4.25, fifteen_min_avg=4.15, cpu_share=3.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 01:59:33.046 7f2b462424e0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 01:59:47.814 7f2b4c9be978 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:00:16.068 7f2b438da198 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:00:23.631 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:00:32.779 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50980300, partition_space_available_1k=30110184, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1378504, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23396, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53904, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14810720, free=9884180, buffers=787564, cached=7182652, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.27, five_min_avg=3.92, fifteen_min_avg=4.04, cpu_share=3.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:01:23.661 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:01:33.791 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50991248, partition_space_available_1k=30099236, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1376208, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23420, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53924, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14884768, free=9787700, buffers=787572, cached=7205076, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.32, five_min_avg=4.35, fifteen_min_avg=4.17, cpu_share=5.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:02:23.620 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:02:27.139 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-458-1', _node_type='Database', _instance_id='0', _timestamp=1734919347139208, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=1, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 12/23/2024 02:02:32.782 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50998804, partition_space_available_1k=30091680, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1380836, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23276, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53920, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14831996, free=9831860, buffers=787704, cached=7213556, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.16, five_min_avg=3.91, fifteen_min_avg=4.03, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:03:23.627 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:03:27.124 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-458-1', _node_type='Database', _instance_id='0', _timestamp=1734919407124040, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 12/23/2024 02:03:32.786 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51003704, partition_space_available_1k=30086780, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1385724, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23364, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53924, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14852380, free=9800664, buffers=788152, cached=7223920, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.65, five_min_avg=3.62, fifteen_min_avg=3.92, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:04:23.636 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:04:29.594 7f2b45de4eb8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:04:32.815 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51009716, partition_space_available_1k=30080768, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1386712, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23492, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53924, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14873504, free=9772952, buffers=788156, cached=7230504, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.15, five_min_avg=3.51, fifteen_min_avg=3.86, cpu_share=3.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:04:43.814 7f2b4d3220f0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:04:49.186 7f2b4d3220f0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:05:23.645 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:05:27.240 7f2b4c11cd68 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:05:32.797 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51019352, partition_space_available_1k=30071132, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1399992, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23588, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53936, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14876044, free=9759756, buffers=788208, cached=7241108, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.72, five_min_avg=3.3, fifteen_min_avg=3.77, cpu_share=3.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:06:24.683 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:06:38.000 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51025612, partition_space_available_1k=30064872, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1393304, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23688, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53932, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14998308, free=9615696, buffers=788232, cached=7262880, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.15, five_min_avg=3.96, fifteen_min_avg=3.94, cpu_share=4.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:07:23.667 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:07:32.794 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51031912, partition_space_available_1k=30058572, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1397792, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23752, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53936, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14889560, free=9717008, buffers=788232, cached=7270316, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.25, five_min_avg=3.73, fifteen_min_avg=3.86, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:08:23.629 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:08:32.796 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51036464, partition_space_available_1k=30054020, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1401912, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23592, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53940, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14901260, free=9699844, buffers=788232, cached=7275780, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.3, five_min_avg=3.52, fifteen_min_avg=3.78, cpu_share=2.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:09:23.644 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:09:31.981 7f2b4607d2e8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:09:32.822 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51042004, partition_space_available_1k=30048480, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1411492, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23676, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54328, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14941568, free=9653112, buffers=788232, cached=7282204, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.39, five_min_avg=3.44, fifteen_min_avg=3.73, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:09:37.335 7f2b46295a20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:09:56.466 7f2b4d3220f0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:10:23.670 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:10:32.826 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51048248, partition_space_available_1k=30042236, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1412760, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23800, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54336, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14912572, free=9670864, buffers=788236, cached=7293444, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.64, five_min_avg=3.52, fifteen_min_avg=3.74, cpu_share=3.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:11:08.984 7f2b4627ea20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:11:23.692 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:11:27.191 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-458-1', _node_type='Database', _instance_id='0', _timestamp=1734919887191405, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=3), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 12/23/2024 02:11:32.836 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51048968, partition_space_available_1k=30041516, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1415912, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23832, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54348, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14898292, free=9683272, buffers=788244, cached=7295308, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.21, five_min_avg=3.48, fifteen_min_avg=3.72, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:12:23.692 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:12:32.849 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51068032, partition_space_available_1k=30022452, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1416908, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23880, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54348, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14936884, free=9623180, buffers=788248, cached=7316804, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.07, five_min_avg=4.25, fifteen_min_avg=3.96, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:13:23.727 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:13:32.863 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51069940, partition_space_available_1k=30020544, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1423832, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23908, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54348, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14982860, free=9565836, buffers=788376, cached=7328044, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.5, five_min_avg=3.83, fifteen_min_avg=3.83, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:13:34.854 7f2b4708ae10 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:13:41.112 7f2b4708ae10 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:14:23.722 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:14:32.888 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51075624, partition_space_available_1k=30014860, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1422496, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23688, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54352, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14940860, free=9599004, buffers=788456, cached=7336796, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.32, five_min_avg=3.76, fifteen_min_avg=3.81, cpu_share=3.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:14:50.718 7f2b438f1978 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:15:23.749 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:15:33.904 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51081232, partition_space_available_1k=30009252, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1436300, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23776, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54348, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14932852, free=9601260, buffers=788464, cached=7342540, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.45, five_min_avg=3.4, fifteen_min_avg=3.68, cpu_share=2.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:16:24.081 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:16:40.077 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51085996, partition_space_available_1k=30004488, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1436412, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23780, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54352, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15056612, free=9456228, buffers=788468, cached=7363808, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=8.1, five_min_avg=4.51, fifteen_min_avg=4.02, cpu_share=4.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:16:46.928 7f2b42887d68 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:17:23.806 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:17:32.914 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51088540, partition_space_available_1k=30001944, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1433852, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23828, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54360, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14967072, free=9530996, buffers=789428, cached=7377620, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.44, five_min_avg=4.07, fifteen_min_avg=3.9, cpu_share=3.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:17:38.603 7f2b443bf0f0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:17:44.352 7f2b47d8fe10 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:18:23.763 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:18:32.913 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51088920, partition_space_available_1k=30001564, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1436716, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23872, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54364, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14989292, free=9505212, buffers=789572, cached=7381040, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.6, five_min_avg=3.6, fifteen_min_avg=3.75, cpu_share=3.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:18:58.505 7f2b43ab96d8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:19:23.766 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:19:32.930 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51094552, partition_space_available_1k=29995932, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1441560, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23916, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54372, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15014168, free=9473464, buffers=789680, cached=7387804, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.22, five_min_avg=3.29, fifteen_min_avg=3.63, cpu_share=2.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:20:23.808 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:20:32.924 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51095632, partition_space_available_1k=29994852, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1444736, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23724, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54372, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14984392, free=9501036, buffers=789684, cached=7390004, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.06, five_min_avg=3.02, fifteen_min_avg=3.51, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:21:23.832 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:21:32.937 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51106572, partition_space_available_1k=29983912, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1448364, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23796, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54376, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15057828, free=9407624, buffers=789692, cached=7409972, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.72, five_min_avg=4.11, fifteen_min_avg=3.85, cpu_share=4.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:22:23.779 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:22:29.132 7f2b43368630 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:22:32.943 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51110376, partition_space_available_1k=29980108, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1451604, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23912, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54376, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15044820, free=9418140, buffers=789692, cached=7412464, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.4, five_min_avg=3.66, fifteen_min_avg=3.71, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:23:17.936 7f2b42891c18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:23:23.641 7f2b43379ac8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:23:23.794 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:23:32.949 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51113268, partition_space_available_1k=29977216, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1455048, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23968, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54384, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15029056, free=9427212, buffers=789692, cached=7419156, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.98, five_min_avg=3.46, fifteen_min_avg=3.64, cpu_share=3.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:24:19.148 7f2b4332cb70 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:24:23.866 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:24:32.956 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51114048, partition_space_available_1k=29976436, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1459748, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23972, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54384, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15031348, free=9423412, buffers=789692, cached=7420664, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.26, five_min_avg=3.16, fifteen_min_avg=3.52, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:25:23.828 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:25:32.951 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51222040, partition_space_available_1k=29868444, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1463356, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23996, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54384, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15020936, free=9425884, buffers=789696, cached=7428600, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.23, five_min_avg=3.02, fifteen_min_avg=3.45, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:26:23.801 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:26:32.955 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51130932, partition_space_available_1k=29959552, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1465920, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23824, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54396, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15056724, free=9472040, buffers=789700, cached=7346652, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.4, five_min_avg=3.83, fifteen_min_avg=3.7, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:26:38.139 7f2b462980f0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:27:23.830 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:27:32.960 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51132380, partition_space_available_1k=29958104, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1469456, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23892, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54392, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15082676, free=9438684, buffers=789700, cached=7354056, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.51, five_min_avg=3.57, fifteen_min_avg=3.62, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:28:23.801 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:28:32.952 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51137272, partition_space_available_1k=29953212, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1473384, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23908, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54396, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15173760, free=9338316, buffers=789700, cached=7363340, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.15, five_min_avg=3.15, fifteen_min_avg=3.47, cpu_share=2.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:28:33.299 7f2b46289ac8 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:28:59.248 7f2b48947438 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:29:05.159 7f2b48947438 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 12/23/2024 02:29:23.816 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:29:32.968 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51138248, partition_space_available_1k=29952236, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1477688, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23956, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54404, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15101940, free=9411412, buffers=789700, cached=7362064, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.44, five_min_avg=3.07, fifteen_min_avg=3.42, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:30:23.843 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:30:33.022 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51144076, partition_space_available_1k=29946408, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1481820, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23980, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54400, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15513092, free=8986472, buffers=789704, cached=7375848, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.96, five_min_avg=3.27, fifteen_min_avg=3.46, cpu_share=3.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:31:23.836 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 12/23/2024 02:31:32.973 7f2b4622b3c8 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-458-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51146916, partition_space_available_1k=29943568, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1494520, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=23988, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=54404, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15122196, free=9371072, buffers=789708, cached=7382140, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.44, five_min_avg=3.71, fifteen_min_avg=3.61, cpu_share=4.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 12/23/2024 02:31:55.020 7f2b46295240 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map