06/19/2025 02:03:09.286 7ff7661dc408 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 06/19/2025 02:03:09.286 7ff7661dc408 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 06/19/2025 02:03:09.286 7ff7661dc408 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 06/19/2025 02:03:09.294 7ff7625886d8 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 06/19/2025 02:03:09.294 7ff7625886d8 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 06/19/2025 02:03:09.295 7ff762591048 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 06/19/2025 02:03:09.295 7ff762591048 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/19/2025 02:03:09.296 7ff762591048 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/19/2025 02:03:09.296 7ff762591048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 06/19/2025 02:03:09.296 7ff762591048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:09.296 7ff762591048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/19/2025 02:03:09.296 7ff762591048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:09.296 7ff762591048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:09.296 7ff762591048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1750298589296081 collector_name = collector_ip = collector_list = [ 10.0.0.130:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 06/19/2025 02:03:09.296 7ff762591048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:09.433 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 4047209511485136750280830746520942808202052921533462909651585323930559341867, 'start': '1750298587000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 80394266889859128975308036735938606414722097775708523330844497999008710104394, 'start': '1750298587000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 33476913203559902136763043090479088438083062183405663558990747853849216347289, 'start': '1750298587000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 06/19/2025 02:03:09.555 7ff76cd06a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:09.556 7ff76cd06a20 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-19 00:46:20.294786", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9585-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 06/19/2025 02:03:09.556 7ff76cd06a20 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:09.556 7ff7661dc408 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3481-1.', _node_type='Database', _instance_id='0', _timestamp=1750298589556645, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/19/2025 02:03:09.556 7ff7661dc408 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3481-1.', _node_type='Database', _instance_id='0', _timestamp=1750298589556924, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1750298587000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1750298587000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1750298587000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-19 00:46:20.294786", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9585-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/19/2025 02:03:09.558 7ff76314fe10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:09.559 7ff76314fe10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/19/2025 02:03:09.559 7ff76314fe10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:09.559 7ff76314fe10 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:09.559 7ff76314fe10 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1750298587000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1750298587000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1750298587000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-19 00:46:20.294786", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9585-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 06/19/2025 02:03:09.559 7ff76314fe10 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:09.933 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/19/2025 02:03:10.286 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 06/19/2025 02:03:10.287 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:10.287 7ff76386f2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. database_usage = [ << disk_space_used_1k = 15300524 disk_space_available_1k = 65789960 analytics_db_size_1k = 124 >>, ] >> 06/19/2025 02:03:10.287 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:10.749 7ff7661dc408 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 06/19/2025 02:03:10.749 7ff7661dc408 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3481-1.', _node_type='Database', _instance_id='0', _timestamp=1750298590749918, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/19/2025 02:03:10.751 7ff763d54780 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:10.751 7ff763d54780 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/19/2025 02:03:10.751 7ff763d54780 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:12.858 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3481-1.', _node_type='Database', _instance_id='0', _timestamp=1750298592858537, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=1), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 06/19/2025 02:03:12.858 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 06/19/2025 02:03:12.860 7ff76386f0f0 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:12.860 7ff76386f0f0 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] CassandraStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. cassandra_compaction_task = << pending_compaction_tasks = 1 >> thread_pool_stats = [ << pool_name = ReadStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = CompactionExecutor active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MutationStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MemtableFlushWriter active = 0 pending = 0 all_time_blocked = 0 >>, ] >> 06/19/2025 02:03:12.860 7ff76386f0f0 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:13.296 7ff765bb4048 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/19/2025 02:03:13.297 7ff765bb4048 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/19/2025 02:03:13.298 7ff765bb4048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/19/2025 02:03:13.298 7ff765bb4048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:13.298 7ff765bb4048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/19/2025 02:03:13.298 7ff765bb4048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:13.298 7ff765bb4048 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/19/2025 02:03:13.299 7ff765bb4048 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/19/2025 02:03:13.299 7ff765bb4048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:13.299 7ff765bb4048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/19/2025 02:03:13.299 7ff765bb4048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:13.299 7ff765bb4048 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:13.299 7ff765bb4048 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1750298589296081 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.130:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/19/2025 02:03:13.299 7ff765bb4048 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:14.044 7ff7661dc408 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 06/19/2025 02:03:17.299 7ff76316b438 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/19/2025 02:03:17.300 7ff76316b438 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/19/2025 02:03:17.300 7ff76316b438 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/19/2025 02:03:17.300 7ff76316b438 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:17.300 7ff76316b438 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/19/2025 02:03:17.300 7ff76316b438 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:17.300 7ff76316b438 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/19/2025 02:03:17.301 7ff76316b438 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/19/2025 02:03:17.301 7ff76316b438 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:17.301 7ff76316b438 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/19/2025 02:03:17.301 7ff76316b438 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:17.301 7ff76316b438 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:17.301 7ff76316b438 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1750298589296081 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.130:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/19/2025 02:03:17.301 7ff76316b438 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:19.067 7ff7661dc408 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3481-1.', _node_type='Database', _instance_id='0', _timestamp=1750298599067418, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1750298587000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1750298587000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1750298587000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-19 00:46:20.294786", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9585-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/19/2025 02:03:19.068 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=15517104, partition_space_available_1k=65573380, percentage_partition_space_used=19, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1218992, cpu_share=0.0, mem_res=1207808, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=8608, cpu_share=0.0, mem_res=6148, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=50040, cpu_share=0.0, mem_res=38552, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665108, used=7791256, free=8405620, buffers=696984, cached=15771248, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.0, five_min_avg=2.15, fifteen_min_avg=1.31, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/19/2025 02:03:19.068 7ff763d5a438 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:19.068 7ff763d5a438 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1750298587000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1750298587000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1750298587000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-19 00:46:20.294786", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9585-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 06/19/2025 02:03:19.068 7ff763d5a438 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:19.068 7ff763d5a438 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:19.069 7ff763d5a438 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 15517104 partition_space_available_1k = 65573380 percentage_partition_space_used = 19 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1218992 cpu_share = 0.0 mem_res = 1207808 >>contrail-query-engine : << mem_virt = 8608 cpu_share = 0.0 mem_res = 6148 >>contrail-database-nodemgr : << mem_virt = 50040 cpu_share = 0.0 mem_res = 38552 >> } system_mem_usage = << total = 32665108 used = 7791256 free = 8405620 buffers = 696984 cached = 15771248 node_type = database-node >> system_cpu_usage = << one_min_avg = 3.0 five_min_avg = 2.15 fifteen_min_avg = 1.31 cpu_share = 0 node_type = database-node >> >> 06/19/2025 02:03:19.069 7ff763d5a438 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:21.301 7ff765f6bf60 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/19/2025 02:03:21.302 7ff765f6bf60 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/19/2025 02:03:21.302 7ff765f6bf60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/19/2025 02:03:21.302 7ff765f6bf60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:21.302 7ff765f6bf60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/19/2025 02:03:21.302 7ff765f6bf60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:21.303 7ff765f6bf60 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/19/2025 02:03:21.303 7ff765f6bf60 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/19/2025 02:03:21.303 7ff765f6bf60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:21.303 7ff765f6bf60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/19/2025 02:03:21.303 7ff765f6bf60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:21.303 7ff765f6bf60 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:21.303 7ff765f6bf60 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1750298589296081 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.130:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/19/2025 02:03:21.303 7ff765f6bf60 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:25.303 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/19/2025 02:03:25.304 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/19/2025 02:03:25.304 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/19/2025 02:03:25.304 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:25.304 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/19/2025 02:03:25.304 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:25.304 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/19/2025 02:03:25.304 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/19/2025 02:03:25.305 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:25.305 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/19/2025 02:03:25.305 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:25.305 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:25.305 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1750298589296081 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.130:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/19/2025 02:03:25.305 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:29.305 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/19/2025 02:03:29.306 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/19/2025 02:03:29.306 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/19/2025 02:03:29.306 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:29.306 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/19/2025 02:03:29.306 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:29.306 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/19/2025 02:03:29.307 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/19/2025 02:03:29.307 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:29.307 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/19/2025 02:03:29.307 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:29.307 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:29.307 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1750298589296081 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.130:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/19/2025 02:03:29.307 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:33.309 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/19/2025 02:03:33.310 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/19/2025 02:03:33.310 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/19/2025 02:03:33.310 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:33.310 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/19/2025 02:03:33.310 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:33.310 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/19/2025 02:03:33.311 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/19/2025 02:03:33.311 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:33.311 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/19/2025 02:03:33.311 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:33.311 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:33.311 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1750298589296081 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.130:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/19/2025 02:03:33.311 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:37.313 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/19/2025 02:03:37.314 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/19/2025 02:03:37.314 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/19/2025 02:03:37.314 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:37.315 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/19/2025 02:03:37.315 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:37.315 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/19/2025 02:03:37.315 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/19/2025 02:03:37.315 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:37.315 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/19/2025 02:03:37.315 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:37.315 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:37.316 7ff763da4d68 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1750298589296081 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.130:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/19/2025 02:03:37.316 7ff763da4d68 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:41.316 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/19/2025 02:03:41.317 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 06/19/2025 02:03:41.317 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/19/2025 02:03:41.317 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:41.317 7ff76386f2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/19/2025 02:03:41.317 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:41.317 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 06/19/2025 02:03:41.318 7ff76386f2e8 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 06/19/2025 02:03:41.318 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 06/19/2025 02:03:41.318 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/19/2025 02:03:41.318 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/19/2025 02:03:49.187 7ff7625afc18 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connection Closed 06/19/2025 02:03:49.188 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 06/19/2025 02:03:49.188 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 06/19/2025 02:03:49.189 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:49.189 7ff76386f2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, ] description = Cassandra state detected DOWN. Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 06/19/2025 02:03:49.189 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:49.189 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:49.189 7ff76386f2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1.:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8103 start_time = 1750298589296081 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.130:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 06/19/2025 02:03:49.190 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/19/2025 02:03:53.192 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/19/2025 02:03:53.193 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Session Event: TCP Connected 06/19/2025 02:03:53.193 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/19/2025 02:03:53.194 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:53.194 7ff76386f2e8 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3481-1. process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/19/2025 02:03:53.194 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/19/2025 02:03:53.194 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 06/19/2025 02:03:53.194 7ff76386f2e8 [contrail-database-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 06/19/2025 02:03:53.195 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 06/19/2025 02:03:53.195 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/19/2025 02:03:53.195 7ff76386f2e8 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/19/2025 02:03:53.206 7ff761b652e8 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001017 06/19/2025 02:03:53.207 7ff761b652e8 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 06/19/2025 02:03:53.209 7ff765c03d68 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 06/19/2025 02:03:53.209 7ff765c03d68 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 06/19/2025 02:03:53.210 7ff765c03d68 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 06/19/2025 02:03:53.210 7ff765c03d68 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 06/19/2025 02:03:53.210 7ff765c03d68 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/19/2025 02:03:53.210 7ff765c03d68 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/19/2025 02:03:53.210 7ff765c03d68 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 06/19/2025 02:03:53.211 7ff761b65eb8 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 06/19/2025 02:03:53.212 7ff761b65eb8 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 06/19/2025 02:03:59.206 7ff761b652e8 [contrail-database-nodemgr] [DEBUG]: Extracted sandesh length: 0000001582 06/19/2025 02:03:59.206 7ff761b652e8 [contrail-database-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 06/19/2025 02:03:59.207 7ff7638f3a20 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 06/19/2025 02:03:59.210 7ff7638f3a20 [contrail-database-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 4 06/19/2025 02:03:59.210 7ff7638f3a20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:DatabaseUsage] map 06/19/2025 02:03:59.212 7ff7638f3a20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 06/19/2025 02:03:59.212 7ff7638f3a20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/19/2025 02:03:59.212 7ff7638f3a20 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/19/2025 02:03:59.213 7ff76c09f048 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 06/19/2025 02:03:59.214 7ff76c09f048 [contrail-database-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 06/19/2025 02:04:09.861 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/19/2025 02:04:10.823 7ff7661dc408 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3481-1.', _node_type='Database', _instance_id='0', _timestamp=1750298650822924, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.130:8086'], status='Up', description='ClientInit to Established on EvSandeshCtrlMessageRecv')]), flag_infos=None, description='')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 06/19/2025 02:04:12.823 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3481-1.', _node_type='Database', _instance_id='0', _timestamp=1750298652823726, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 06/19/2025 02:04:17.845 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=16869224, partition_space_available_1k=64221260, percentage_partition_space_used=21, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1302628, cpu_share=0.33, mem_res=1285996, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10308, cpu_share=0.0, mem_res=7704, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=50256, cpu_share=0.02, mem_res=38728, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665108, used=9018940, free=5676884, buffers=725640, cached=17243644, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.85, five_min_avg=2.49, fifteen_min_avg=1.47, cpu_share=4.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/19/2025 02:05:09.933 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/19/2025 02:05:18.175 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=17558996, partition_space_available_1k=63531488, percentage_partition_space_used=22, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1306780, cpu_share=0.1, mem_res=1284936, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10084, cpu_share=0.0, mem_res=7872, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41076, cpu_share=0.03, mem_res=38756, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665108, used=8976060, free=20364092, buffers=740476, cached=2584480, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.37, five_min_avg=2.62, fifteen_min_avg=1.58, cpu_share=4.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/19/2025 02:05:41.741 7ff763171e10 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/19/2025 02:05:43.800 7ff763da40f0 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/19/2025 02:06:09.880 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/19/2025 02:06:17.858 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=18703804, partition_space_available_1k=62386680, percentage_partition_space_used=23, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1310380, cpu_share=0.11, mem_res=1293404, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=10404, cpu_share=0.0, mem_res=8188, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41076, cpu_share=0.03, mem_res=38764, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665108, used=9583992, free=18056272, buffers=770088, cached=4254756, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.49, five_min_avg=2.83, fifteen_min_avg=1.72, cpu_share=3.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/19/2025 02:06:43.021 7ff761b65828 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/19/2025 02:07:09.994 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/19/2025 02:07:14.304 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3481-1.', _node_type='Database', _instance_id='0', _timestamp=1750298834304719, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=22), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 06/19/2025 02:07:19.328 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19125004, partition_space_available_1k=61965480, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1374476, cpu_share=0.21, mem_res=1350400, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13332, cpu_share=0.0, mem_res=11036, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41092, cpu_share=0.03, mem_res=38768, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665108, used=10376248, free=17190352, buffers=779612, cached=4318896, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.52, five_min_avg=3.64, fifteen_min_avg=2.07, cpu_share=4.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/19/2025 02:08:09.961 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/19/2025 02:08:18.052 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19155372, partition_space_available_1k=61935112, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1357820, cpu_share=0.16, mem_res=1329088, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13440, cpu_share=0.0, mem_res=11148, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41100, cpu_share=0.03, mem_res=38768, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665108, used=10287736, free=17213688, buffers=780028, cached=4383656, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.54, five_min_avg=3.57, fifteen_min_avg=2.14, cpu_share=4.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/19/2025 02:09:09.900 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/19/2025 02:09:12.785 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3481-1.', _node_type='Database', _instance_id='0', _timestamp=1750298952785641, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=23), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 06/19/2025 02:09:17.809 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19227404, partition_space_available_1k=61863080, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1367704, cpu_share=0.14, mem_res=1334564, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13524, cpu_share=0.0, mem_res=11232, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41108, cpu_share=0.03, mem_res=38768, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665108, used=11189456, free=16261392, buffers=780136, cached=4434124, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.98, five_min_avg=4.11, fifteen_min_avg=2.41, cpu_share=5.38, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/19/2025 02:10:09.911 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/19/2025 02:10:17.841 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19248072, partition_space_available_1k=61842412, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1396428, cpu_share=0.13, mem_res=1359708, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13428, cpu_share=0.0, mem_res=11136, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41112, cpu_share=0.03, mem_res=38768, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665108, used=10647708, free=16770508, buffers=780192, cached=4466700, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.54, five_min_avg=3.99, fifteen_min_avg=2.47, cpu_share=3.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/19/2025 02:11:09.902 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/19/2025 02:11:17.759 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19273052, partition_space_available_1k=61817432, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1381764, cpu_share=0.11, mem_res=1341932, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13480, cpu_share=0.0, mem_res=11188, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41112, cpu_share=0.03, mem_res=38768, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665108, used=10495828, free=16895632, buffers=780392, cached=4493256, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.24, five_min_avg=3.71, fifteen_min_avg=2.47, cpu_share=4.12, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/19/2025 02:12:09.988 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/19/2025 02:12:18.061 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19304272, partition_space_available_1k=61786212, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1388104, cpu_share=0.11, mem_res=1344948, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13556, cpu_share=0.0, mem_res=11264, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41128, cpu_share=0.03, mem_res=38768, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665108, used=11382724, free=15977264, buffers=780428, cached=4524692, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.88, five_min_avg=3.5, fifteen_min_avg=2.47, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/19/2025 02:12:41.366 7ff763175438 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 06/19/2025 02:13:09.909 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 06/19/2025 02:13:17.730 7ff7661dc408 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3481-1.', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19456440, partition_space_available_1k=61634044, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1395484, cpu_share=0.09, mem_res=1349952, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13724, cpu_share=0.0, mem_res=11428, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41120, cpu_share=0.02, mem_res=38768, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665108, used=10383660, free=16801756, buffers=780512, cached=4699180, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.07, five_min_avg=3.49, fifteen_min_avg=2.54, cpu_share=4.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 06/19/2025 02:13:34.203 7ff763d8bc18 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map