10/21/2024 02:01:27.468 7fd1179c4f20 [contrail-database-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 10/21/2024 02:01:27.468 7fd1179c4f20 [contrail-database-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 10/21/2024 02:01:27.468 7fd1179c4f20 [contrail-database-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 10/21/2024 02:01:27.478 7fd1112d6248 [contrail-database-nodemgr] [ERROR]: Cannot write http_port 8103 to /tmp/contrail-database-nodemgr.0.http_port 10/21/2024 02:01:27.478 7fd1112d6248 [contrail-database-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8103 10/21/2024 02:01:27.479 7fd1112d6448 [contrail-database-nodemgr] [INFO]: Processing event[EvStart] in state[none] 10/21/2024 02:01:27.480 7fd1112d6448 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:01:27.481 7fd1112d6448 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 10/21/2024 02:01:27.481 7fd1112d6448 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 10/21/2024 02:01:27.487 7fd1112d6448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:27.489 7fd1112d6448 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 10/21/2024 02:01:27.489 7fd1112d6448 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:27.489 7fd1112d6448 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:27.489 7fd1112d6448 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 10/21/2024 02:01:27.489 7fd1112d6448 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:27.929 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'cassandra', 'group': 'cassandra', 'pid': 97716801164097403901711857022253310083866925107670519481339987274389454091643, 'start': '1729476083000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-query-engine', 'group': 'contrail-query-engine', 'pid': 28739542013191634464567353250803988017614376828812083364861753228302987583535, 'start': '1729476084000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-database-nodemgr', 'group': 'contrail-database-nodemgr', 'pid': 105086381439152950967657693784892032549793890278979712192326968548457741235058, 'start': '1729476084000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 10/21/2024 02:01:28.008 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:01:28.137 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:28.137 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2024-10-21 01:11:05.701862", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-6068-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 10/21/2024 02:01:28.138 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:28.138 7fd1179c4f20 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-439-1', _node_type='Database', _instance_id='0', _timestamp=1729476088138643, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 10/21/2024 02:01:28.139 7fd1179c4f20 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-439-1', _node_type='Database', _instance_id='0', _timestamp=1729476088139167, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1729476083000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1729476084000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1729476084000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2024-10-21 01:11:05.701862", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-6068-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 10/21/2024 02:01:28.153 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:28.153 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 10/21/2024 02:01:28.154 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:28.154 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:28.154 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1729476083000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1729476084000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1729476084000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2024-10-21 01:11:05.701862", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-6068-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 10/21/2024 02:01:28.154 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:28.958 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:01:29.966 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:DatabaseUsage] map 10/21/2024 02:01:29.969 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:29.970 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51331756 disk_space_available_1k = 29758728 analytics_db_size_1k = 4 >>, ] >> 10/21/2024 02:01:29.970 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:30.874 7fd1179c4f20 [contrail-database-nodemgr] [ERROR]: Failed to connect to database by CQL: Result: 1 10/21/2024 02:01:30.875 7fd1179c4f20 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-439-1', _node_type='Database', _instance_id='0', _timestamp=1729476090875066, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 10/21/2024 02:01:30.879 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:30.879 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Cassandra state detected DOWN. Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 10/21/2024 02:01:30.880 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:31.481 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:01:33.513 7fd1179c4f20 [contrail-database-nodemgr] [ERROR]: Failed to get nodetool compactionstats: Result: 2 10/21/2024 02:01:36.287 7fd1179c4f20 [contrail-database-nodemgr] [ERROR]: Failed to run cmd: 'nodetool -p 7200 repair -pr ContrailAnalyticsCql'. Error is: Result: 1 10/21/2024 02:01:36.484 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:01:36.485 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:01:36.493 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:01:36.494 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:01:36.494 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:01:36.494 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:01:36.495 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:01:36.496 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:36.496 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:01:36.496 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:36.496 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:36.496 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 10/21/2024 02:01:36.496 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:40.485 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:01:41.339 7fd1179c4f20 [contrail-database-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-439-1', _node_type='Database', _instance_id='0', _timestamp=1729476101339249, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='cassandra', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1729476083000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-query-engine', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1729476084000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-database-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1729476084000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2024-10-21 01:11:05.701862", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-6068-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 10/21/2024 02:01:41.340 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51339300, partition_space_available_1k=29751184, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1268896, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=11600, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=53532, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14376204, free=593088, buffers=568112, cached=17127712, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=15.5, five_min_avg=6.93, fifteen_min_avg=3.86, cpu_share=0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:01:45.341 7fd1113d8848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:01:45.488 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:01:45.489 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:01:45.489 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:01:45.489 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:01:45.490 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:01:45.490 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:01:45.490 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 deleted = False process_info = [ << process_name = cassandra process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1729476083000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-query-engine process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1729476084000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-database-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1729476084000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2024-10-21 01:11:05.701862", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-6068-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 10/21/2024 02:01:45.490 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:01:45.490 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:01:45.490 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51339300 partition_space_available_1k = 29751184 percentage_partition_space_used = 63 >> } all_core_file_list = [ ] process_mem_cpu_usage = { cassandra : << mem_virt = 1268896 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 11600 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 53532 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14376204 free = 593088 buffers = 568112 cached = 17127712 node_type = database-node >> system_cpu_usage = << one_min_avg = 15.5 five_min_avg = 6.93 fifteen_min_avg = 3.86 cpu_share = 0 node_type = database-node >> >> 10/21/2024 02:01:45.490 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:01:45.491 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:01:45.491 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:01:45.491 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:45.492 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:01:45.492 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:45.492 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:45.492 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:01:45.492 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:49.489 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:01:54.498 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:01:54.498 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:01:54.499 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:01:54.499 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:01:54.500 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:01:54.500 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:01:54.501 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:01:54.501 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:54.501 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:01:54.501 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:54.501 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:54.502 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:01:54.502 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:01:58.501 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:02:03.507 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:02:03.507 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:02:03.507 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:03.508 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:02:03.508 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:03.508 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:02:03.508 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:02:03.508 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:03.509 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:02:03.509 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:03.509 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:03.509 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:02:03.509 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:06.066 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:02:07.507 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:02:12.513 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:02:12.513 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:02:12.513 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:12.513 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:02:12.514 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:12.514 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:02:12.514 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:02:12.514 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:12.514 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:02:12.514 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:12.515 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:12.515 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:02:12.515 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:16.514 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:02:21.522 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:02:21.522 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:02:21.522 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:21.522 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:02:21.522 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:21.523 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:02:21.523 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:02:21.524 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:21.524 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Cassandra state detected DOWN. Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:02:21.524 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:21.524 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:21.526 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:02:21.527 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:25.525 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:02:28.489 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:02:29.844 7fd1179c4f20 [contrail-database-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-439-1', _node_type='Database', _instance_id='0', _timestamp=1729476149844080, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-database-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=['10.0.0.30:8086'], status='Initializing', description='Idle to Connect on EvIdleHoldTimerExpired')]), flag_infos=None, description='Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo'), _seqnum=0) 10/21/2024 02:02:30.530 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:02:30.531 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:02:30.531 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:30.531 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Cassandra state detected DOWN. Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:02:30.531 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:30.531 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:30.531 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51388712 disk_space_available_1k = 29701772 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:02:30.532 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:30.532 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:30.532 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:02:30.532 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:30.532 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:02:30.532 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:02:30.532 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:30.533 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:02:30.533 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:30.533 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:30.533 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:02:30.533 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:32.836 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: Sending UVE: CassandraStatusUVE(_context='', _scope='', _module='contrail-database-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-439-1', _node_type='Database', _instance_id='0', _timestamp=1729476152836761, _versionsig=46427434, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=CassandraStatusData(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, cassandra_compaction_task=CassandraCompactionTask(pending_compaction_tasks=0), thread_pool_stats=[CassandraThreadPoolStats(pool_name='ReadStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='CompactionExecutor', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MutationStage', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='Native-Transport-Requests', active=0, pending=0, all_time_blocked=0), CassandraThreadPoolStats(pool_name='MemtableFlushWriter', active=0, pending=0, all_time_blocked=0)], _table='ObjectDatabaseInfo'), _seqnum=0) 10/21/2024 02:02:32.837 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: Add uve in the [ObjectDatabaseInfo:CassandraStatusUVE] map 10/21/2024 02:02:32.841 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:32.842 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] CassandraStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 cassandra_compaction_task = << pending_compaction_tasks = 0 >> thread_pool_stats = [ << pool_name = ReadStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = CompactionExecutor active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MutationStage active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = Native-Transport-Requests active = 0 pending = 0 all_time_blocked = 0 >>, << pool_name = MemtableFlushWriter active = 0 pending = 0 all_time_blocked = 0 >>, ] >> 10/21/2024 02:02:32.843 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:34.531 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:02:37.874 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51391428, partition_space_available_1k=29699056, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1312544, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=12856, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=45844, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14567628, free=11889284, buffers=613836, cached=5594368, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=13.1, five_min_avg=7.79, fifteen_min_avg=4.35, cpu_share=8.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:02:39.535 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:02:39.535 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:02:39.535 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:39.535 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:02:39.536 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:39.536 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:39.536 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51391428 partition_space_available_1k = 29699056 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1312544 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 12856 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 45844 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14567628 free = 11889284 buffers = 613836 cached = 5594368 node_type = database-node >> system_cpu_usage = << one_min_avg = 13.1 five_min_avg = 7.79 fifteen_min_avg = 4.35 cpu_share = 8.0 node_type = database-node >> >> 10/21/2024 02:02:39.536 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:39.536 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:02:39.537 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:02:39.537 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:39.537 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:02:39.537 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:39.537 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:39.537 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:02:39.537 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:39.903 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:02:43.537 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:02:48.541 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:02:48.541 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:02:48.541 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:48.541 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:02:48.542 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:48.542 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:02:48.542 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:02:48.542 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:48.542 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:02:48.543 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:48.544 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:48.544 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:02:48.544 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:52.542 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:02:57.550 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:02:57.550 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:02:57.550 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:57.551 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:02:57.551 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:02:57.551 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:02:57.551 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:02:57.551 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:57.551 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:02:57.552 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:57.552 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:02:57.552 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:02:57.552 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:01.550 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:03:06.556 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:03:06.556 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:03:06.557 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:03:06.557 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:03:06.557 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:03:06.557 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:03:06.557 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:03:06.558 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:06.558 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:03:06.558 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:06.558 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:06.558 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:03:06.558 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:10.559 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:03:11.104 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:03:15.562 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:03:15.563 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:03:15.563 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:03:15.563 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:03:15.563 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:03:15.564 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:03:15.564 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:03:15.565 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:15.565 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:03:15.565 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:15.565 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:15.565 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:03:15.565 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:19.566 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:03:24.569 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:03:24.569 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:03:24.569 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:03:24.569 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:03:24.570 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:03:24.570 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:03:24.570 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:03:24.570 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:24.570 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:03:24.570 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:24.571 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:24.571 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:03:24.571 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:25.261 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:03:28.467 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:03:28.574 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:03:33.580 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:03:33.581 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:03:33.581 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:03:33.582 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:03:33.582 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:03:33.582 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:03:33.582 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51437384 disk_space_available_1k = 29653100 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:03:33.583 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:03:33.583 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:03:33.584 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:03:33.584 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:33.584 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:03:33.584 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:33.584 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:33.585 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:03:33.585 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:37.144 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51439108, partition_space_available_1k=29651376, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1297028, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13068, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=45936, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14685684, free=11597852, buffers=621784, cached=5759796, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=7.5, five_min_avg=7.21, fifteen_min_avg=4.37, cpu_share=5.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:03:37.144 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:37.145 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51439108 partition_space_available_1k = 29651376 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1297028 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 13068 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 45936 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14685684 free = 11597852 buffers = 621784 cached = 5759796 node_type = database-node >> system_cpu_usage = << one_min_avg = 7.5 five_min_avg = 7.21 fifteen_min_avg = 4.37 cpu_share = 5.0 node_type = database-node >> >> 10/21/2024 02:03:37.145 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:37.581 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:03:42.586 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:03:42.586 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:03:42.587 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:03:42.587 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:03:42.587 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:03:42.587 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:03:42.588 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:03:42.588 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:42.588 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:03:42.588 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:42.588 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:42.589 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:03:42.589 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:46.586 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:03:51.589 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:03:51.589 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:03:51.589 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:03:51.590 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:03:51.590 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:03:51.590 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:03:51.590 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:03:51.591 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:51.591 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:03:51.591 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:51.591 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:51.591 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:03:51.591 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:03:55.591 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:04:00.593 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:04:00.593 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:04:00.593 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:04:00.594 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:04:00.594 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:04:00.594 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:04:00.594 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:04:00.595 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:00.595 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:04:00.595 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:00.595 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:00.595 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:04:00.595 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:04.596 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:04:09.602 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:04:09.602 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:04:09.602 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:04:09.602 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:04:09.602 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:04:09.602 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:04:09.603 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:04:09.603 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:09.603 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:04:09.603 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:09.603 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:09.604 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:04:09.604 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:13.605 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:04:18.608 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:04:18.609 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:04:18.609 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:04:18.609 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:04:18.609 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:04:18.609 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:04:18.610 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:04:18.610 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:18.610 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:04:18.610 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:18.610 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:18.611 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:04:18.611 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:22.611 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:04:27.616 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:04:27.616 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:04:27.616 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:04:27.616 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:04:27.617 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:04:27.617 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:04:27.617 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:04:27.617 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:27.617 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:04:27.618 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:27.618 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:27.618 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:04:27.618 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:28.449 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:04:28.832 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:28.832 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51431964 disk_space_available_1k = 29658520 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:04:28.832 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:31.616 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:04:36.619 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:04:36.619 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:04:36.619 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:04:36.620 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:04:36.620 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:04:36.620 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:04:36.621 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:04:36.621 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:36.621 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:04:36.621 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:36.621 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:36.621 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:04:36.621 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:36.929 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51432160, partition_space_available_1k=29658324, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1298732, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13280, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46264, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14594316, free=11684616, buffers=622944, cached=5763240, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.1, five_min_avg=6.52, fifteen_min_avg=4.3, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:04:36.929 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:36.930 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51432160 partition_space_available_1k = 29658324 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1298732 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 13280 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46264 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14594316 free = 11684616 buffers = 622944 cached = 5763240 node_type = database-node >> system_cpu_usage = << one_min_avg = 5.1 five_min_avg = 6.52 fifteen_min_avg = 4.3 cpu_share = 2.62 node_type = database-node >> >> 10/21/2024 02:04:36.930 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:40.621 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:04:44.306 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:04:45.624 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:04:45.625 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:04:45.625 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:04:45.625 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:04:45.625 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:04:45.626 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:04:45.627 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:04:45.627 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:45.627 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:04:45.628 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:45.628 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:45.628 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:04:45.628 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:49.627 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:04:54.633 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:04:54.633 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:04:54.633 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:04:54.633 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:04:54.634 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:04:54.634 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:04:54.634 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:04:54.634 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:54.634 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:04:54.634 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:54.635 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:54.635 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:04:54.635 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:04:58.636 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:04:59.560 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:05:03.639 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:05:03.640 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:05:03.640 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:03.640 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:05:03.640 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:03.641 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:05:03.641 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:05:03.641 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:03.642 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:05:03.642 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:03.642 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:03.642 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:05:03.642 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:07.642 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:05:12.645 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:05:12.645 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:05:12.645 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:12.645 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:05:12.645 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:12.646 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:05:12.646 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:05:12.646 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:12.646 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:05:12.646 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:12.646 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:12.647 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:05:12.647 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:13.649 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:05:16.645 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:05:21.650 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:05:21.650 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:05:21.650 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:21.650 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:05:21.651 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:21.651 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:05:21.651 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:05:21.652 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:21.652 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:05:21.652 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:21.652 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:21.652 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:05:21.652 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:25.653 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:05:28.531 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:05:30.658 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:05:30.659 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:05:30.659 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:30.659 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:05:30.659 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:30.659 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:30.659 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51436680 disk_space_available_1k = 29653804 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:05:30.660 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:30.660 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:05:30.660 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:05:30.660 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:30.661 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:05:30.661 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:30.661 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:30.661 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:05:30.661 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:34.659 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:05:36.804 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51436856, partition_space_available_1k=29653628, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1300712, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13468, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46312, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14641216, free=11555356, buffers=626064, cached=5842480, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.13, five_min_avg=6.08, fifteen_min_avg=4.29, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:05:39.664 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:05:39.664 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:05:39.664 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:39.664 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:05:39.664 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:39.665 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:39.665 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51436856 partition_space_available_1k = 29653628 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1300712 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 13468 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46312 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14641216 free = 11555356 buffers = 626064 cached = 5842480 node_type = database-node >> system_cpu_usage = << one_min_avg = 4.13 five_min_avg = 6.08 fifteen_min_avg = 4.29 cpu_share = 2.62 node_type = database-node >> >> 10/21/2024 02:05:39.665 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:39.665 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:05:39.665 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:05:39.666 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:39.666 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:05:39.666 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:39.666 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:39.666 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:05:39.666 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:43.665 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:05:48.668 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:05:48.668 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:05:48.668 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:48.669 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:05:48.669 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:48.669 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:05:48.669 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:05:48.669 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:48.670 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:05:48.670 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:48.670 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:48.670 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:05:48.670 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:52.669 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:05:53.572 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:05:57.674 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:05:57.674 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:05:57.674 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:57.675 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:05:57.675 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:05:57.675 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:05:57.675 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:05:57.675 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:57.676 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:05:57.676 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:57.676 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:05:57.676 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:05:57.676 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:01.684 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:06:06.694 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:06:06.694 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:06:06.695 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:06:06.695 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:06:06.695 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:06:06.695 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:06:06.695 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:06:06.696 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:06.696 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:06:06.696 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:06.696 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:06.696 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:06:06.696 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:10.697 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:06:15.702 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:06:15.702 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:06:15.702 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:06:15.702 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:06:15.703 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:06:15.703 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:06:15.703 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:06:15.703 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:15.704 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:06:15.704 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:15.704 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:15.704 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:06:15.704 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:19.704 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:06:24.710 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:06:24.710 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:06:24.710 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:06:24.710 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:06:24.711 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:06:24.711 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:06:24.711 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:06:24.711 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:24.711 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:06:24.711 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:24.712 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:24.712 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:06:24.712 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:28.479 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:06:28.715 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:06:33.720 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:06:33.720 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:06:33.720 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:06:33.721 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:06:33.721 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:06:33.721 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:06:33.721 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51449092 disk_space_available_1k = 29641392 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:06:33.721 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:06:33.721 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:06:33.722 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:06:33.722 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:33.722 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:06:33.722 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:33.722 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:33.722 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:06:33.723 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:36.882 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51449256, partition_space_available_1k=29641228, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1331256, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13672, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46356, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14648876, free=10238720, buffers=678360, cached=7099160, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=8.55, five_min_avg=6.94, fifteen_min_avg=4.69, cpu_share=5.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:06:36.883 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:36.883 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51449256 partition_space_available_1k = 29641228 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1331256 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 13672 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46356 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14648876 free = 10238720 buffers = 678360 cached = 7099160 node_type = database-node >> system_cpu_usage = << one_min_avg = 8.55 five_min_avg = 6.94 fifteen_min_avg = 4.69 cpu_share = 5.5 node_type = database-node >> >> 10/21/2024 02:06:36.883 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:37.720 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:06:42.725 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:06:42.725 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:06:42.725 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:06:42.725 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:06:42.725 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:06:42.726 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:06:42.726 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:06:42.726 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:42.726 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:06:42.727 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:42.728 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:42.728 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:06:42.728 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:46.726 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:06:51.729 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:06:51.729 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:06:51.729 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:06:51.730 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:06:51.730 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:06:51.730 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:06:51.730 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:06:51.730 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:51.730 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:06:51.731 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:51.731 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:51.731 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:06:51.731 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:06:55.732 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:07:00.738 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:07:00.738 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:07:00.738 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:07:00.739 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:07:00.739 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:07:00.739 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:07:00.739 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:07:00.740 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:00.740 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:07:00.741 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:00.741 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:00.741 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:07:00.741 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:04.739 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:07:09.743 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:07:09.744 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:07:09.744 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:07:09.744 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:07:09.744 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:07:09.744 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:07:09.745 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:07:09.745 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:09.745 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:07:09.745 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:09.746 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:09.746 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:07:09.746 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:13.747 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:07:18.749 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:07:18.749 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:07:18.749 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:07:18.750 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:07:18.750 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:07:18.750 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:07:18.750 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:07:18.751 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:18.751 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:07:18.751 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:18.751 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:18.751 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:07:18.751 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:22.751 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:07:27.757 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:07:27.757 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:07:27.757 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:07:27.758 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:07:27.758 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:07:27.758 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:07:27.759 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:07:27.759 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:27.759 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:07:27.759 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:27.760 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:27.760 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:07:27.760 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:28.471 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:07:28.875 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:28.876 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51452848 disk_space_available_1k = 29637636 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:07:28.876 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:31.758 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:07:36.762 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:07:36.762 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:07:36.762 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:07:36.762 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:07:36.763 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:07:36.763 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:07:36.763 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:07:36.764 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:36.764 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:07:36.764 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:36.764 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:36.764 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:07:36.764 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:36.990 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51453044, partition_space_available_1k=29637440, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1306732, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13880, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46404, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14649344, free=10232944, buffers=678364, cached=7104464, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.29, five_min_avg=6.04, fifteen_min_avg=4.52, cpu_share=2.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:07:36.991 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:36.991 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51453044 partition_space_available_1k = 29637440 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1306732 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 13880 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46404 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14649344 free = 10232944 buffers = 678364 cached = 7104464 node_type = database-node >> system_cpu_usage = << one_min_avg = 4.29 five_min_avg = 6.04 fifteen_min_avg = 4.52 cpu_share = 2.5 node_type = database-node >> >> 10/21/2024 02:07:36.991 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:40.763 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:07:45.768 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:07:45.768 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:07:45.769 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:07:45.769 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:07:45.769 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:07:45.769 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:07:45.770 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:07:45.770 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:45.770 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:07:45.771 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:45.771 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:45.771 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:07:45.771 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:49.769 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:07:54.773 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:07:54.774 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:07:54.774 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:07:54.774 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:07:54.774 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:07:54.774 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:07:54.775 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:07:54.775 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:54.776 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:07:54.776 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:54.776 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:54.776 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:07:54.776 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:07:58.776 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:08:03.778 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:08:03.779 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:08:03.779 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:03.779 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:08:03.779 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:03.779 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:08:03.780 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:08:03.780 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:03.780 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:08:03.780 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:03.780 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:03.780 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:08:03.780 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:07.782 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:08:12.786 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:08:12.787 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:08:12.787 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:12.787 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:08:12.787 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:12.787 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:08:12.788 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:08:12.788 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:12.788 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:08:12.788 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:12.788 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:12.788 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:08:12.788 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:16.788 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:08:21.792 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:08:21.792 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:08:21.793 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:21.793 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:08:21.793 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:21.793 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:08:21.794 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:08:21.794 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:21.794 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:08:21.794 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:21.794 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:21.794 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:08:21.795 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:25.795 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:08:28.445 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:08:30.797 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:08:30.797 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:08:30.797 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:30.797 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:08:30.798 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:30.798 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:30.798 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51456472 disk_space_available_1k = 29634012 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:08:30.798 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:30.798 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:08:30.799 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:08:30.800 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:30.801 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:08:30.801 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:30.801 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:30.801 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:08:30.801 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:34.798 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:08:36.832 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51456692, partition_space_available_1k=29633792, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1306728, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14108, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46460, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14655268, free=10221068, buffers=678372, cached=7110408, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.07, five_min_avg=5.38, fifteen_min_avg=4.39, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:08:39.801 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:08:39.801 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:08:39.801 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:39.801 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:08:39.802 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:39.802 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:39.802 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51456692 partition_space_available_1k = 29633792 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1306728 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 14108 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46460 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14655268 free = 10221068 buffers = 678372 cached = 7110408 node_type = database-node >> system_cpu_usage = << one_min_avg = 3.07 five_min_avg = 5.38 fifteen_min_avg = 4.39 cpu_share = 2.62 node_type = database-node >> >> 10/21/2024 02:08:39.802 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:39.802 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:08:39.803 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:08:39.803 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:39.803 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:08:39.803 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:39.803 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:39.803 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:08:39.803 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:43.805 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:08:48.807 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:08:48.807 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:08:48.807 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:48.808 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:08:48.808 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:48.808 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:08:48.809 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:08:48.809 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:48.809 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:08:48.809 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:48.810 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:48.810 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:08:48.810 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:52.810 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:08:57.816 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:08:57.816 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:08:57.816 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:57.816 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:08:57.817 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:08:57.817 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:08:57.817 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:08:57.817 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:57.817 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:08:57.817 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:57.818 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:57.818 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:08:57.818 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:08:58.697 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:09:01.818 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:09:06.821 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:09:06.822 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:09:06.822 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:09:06.822 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:09:06.822 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:09:06.823 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:09:06.823 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:09:06.823 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:06.823 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:09:06.823 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:06.824 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:06.824 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:09:06.824 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:10.825 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:09:12.603 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:09:15.828 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:09:15.828 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:09:15.828 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:09:15.829 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:09:15.829 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:09:15.829 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:09:15.829 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:09:15.829 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:15.830 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:09:15.830 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:15.830 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:15.830 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:09:15.830 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:19.828 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:09:24.833 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:09:24.833 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:09:24.833 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:09:24.833 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:09:24.833 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:09:24.833 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:09:24.834 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:09:24.834 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:24.834 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:09:24.834 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:24.835 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:24.835 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:09:24.835 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:28.479 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:09:28.833 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:09:29.597 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:09:33.837 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:09:33.837 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:09:33.838 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:09:33.838 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:09:33.838 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:09:33.838 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:09:33.838 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51463584 disk_space_available_1k = 29626900 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:09:33.838 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:09:33.838 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:09:33.839 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:09:33.839 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:33.839 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:09:33.839 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:33.839 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:33.840 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:09:33.840 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:37.041 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51464928, partition_space_available_1k=29625556, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1326396, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14260, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46524, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14718576, free=10148316, buffers=678464, cached=7119760, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.5, five_min_avg=4.77, fifteen_min_avg=4.24, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:09:37.041 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:37.041 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51464928 partition_space_available_1k = 29625556 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1326396 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 14260 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46524 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14718576 free = 10148316 buffers = 678464 cached = 7119760 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.5 five_min_avg = 4.77 fifteen_min_avg = 4.24 cpu_share = 2.75 node_type = database-node >> >> 10/21/2024 02:09:37.041 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:37.838 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:09:42.843 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:09:42.844 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:09:42.844 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:09:42.844 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:09:42.844 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:09:42.844 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:09:42.845 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:09:42.845 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:42.845 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:09:42.845 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:42.845 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:42.845 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:09:42.845 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:46.846 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:09:51.849 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:09:51.849 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:09:51.850 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:09:51.850 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:09:51.850 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:09:51.850 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:09:51.850 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:09:51.851 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:51.851 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:09:51.851 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:51.851 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:51.851 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:09:51.851 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:09:54.414 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:09:55.850 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:10:00.855 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:10:00.855 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:10:00.855 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:10:00.855 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:10:00.855 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:10:00.856 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:10:00.856 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:10:00.856 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:00.856 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:10:00.856 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:00.857 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:00.857 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:10:00.857 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:04.856 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:10:09.861 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:10:09.862 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:10:09.862 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:10:09.862 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:10:09.863 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:10:09.863 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:10:09.863 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:10:09.864 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:09.864 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:10:09.864 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:09.864 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:09.864 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:10:09.865 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:13.865 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:10:18.871 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:10:18.871 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:10:18.871 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:10:18.871 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:10:18.871 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:10:18.872 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:10:18.872 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:10:18.872 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:18.872 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:10:18.873 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:18.873 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:18.873 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:10:18.873 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:22.873 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:10:27.878 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:10:27.879 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:10:27.879 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:10:27.879 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:10:27.879 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:10:27.879 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:10:27.880 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:10:27.880 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:27.881 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:10:27.881 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:27.881 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:27.881 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:10:27.881 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:28.543 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:10:28.950 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:28.951 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51464472 disk_space_available_1k = 29626012 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:10:28.951 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:31.879 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:10:36.881 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:10:36.882 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:10:36.882 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:10:36.882 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:10:36.883 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:10:36.883 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:10:36.884 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:10:36.884 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:36.884 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:10:36.884 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:36.885 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:36.885 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:10:36.885 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:37.430 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51464780, partition_space_available_1k=29625704, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1305604, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14472, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46568, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14646608, free=10218464, buffers=678476, cached=7121568, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.77, five_min_avg=4.42, fifteen_min_avg=4.15, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:10:37.431 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:37.431 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51464780 partition_space_available_1k = 29625704 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1305604 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 14472 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46568 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14646608 free = 10218464 buffers = 678476 cached = 7121568 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.77 five_min_avg = 4.42 fifteen_min_avg = 4.15 cpu_share = 2.75 node_type = database-node >> >> 10/21/2024 02:10:37.431 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:40.884 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:10:45.889 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:10:45.890 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:10:45.890 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:10:45.890 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:10:45.890 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:10:45.891 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:10:45.891 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:10:45.891 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:45.892 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:10:45.892 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:45.892 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:45.892 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:10:45.892 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:49.889 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:10:54.894 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:10:54.894 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:10:54.894 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:10:54.894 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:10:54.894 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:10:54.894 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:10:54.895 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:10:54.895 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:54.895 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:10:54.895 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:54.896 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:54.896 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:10:54.896 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:10:58.897 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:11:03.901 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:11:03.901 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:11:03.901 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:03.901 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:11:03.902 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:03.902 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:11:03.902 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:11:03.902 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:03.903 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:11:03.903 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:03.903 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:03.903 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:11:03.903 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:07.901 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:11:12.906 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:11:12.906 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:11:12.906 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:12.907 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:11:12.907 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:12.907 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:11:12.907 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:11:12.907 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:12.908 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:11:12.908 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:12.908 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:12.908 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:11:12.908 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:16.909 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:11:21.911 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:11:21.912 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:11:21.912 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:21.912 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:11:21.912 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:21.912 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:11:21.913 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:11:21.913 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:21.913 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:11:21.913 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:21.913 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:21.914 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:11:21.914 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:25.913 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:11:28.466 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:11:30.915 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:11:30.916 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:11:30.916 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:30.916 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:11:30.916 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:30.916 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:30.917 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51468608 disk_space_available_1k = 29621876 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:11:30.917 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:30.917 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:11:30.917 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:11:30.918 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:30.918 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:11:30.918 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:30.918 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:30.918 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:11:30.918 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:34.916 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:11:36.824 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51468888, partition_space_available_1k=29621596, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1305708, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14668, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46612, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14648772, free=10210092, buffers=678476, cached=7127776, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.56, five_min_avg=4.06, fifteen_min_avg=4.04, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:11:39.919 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:11:39.920 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:11:39.920 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:39.920 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:11:39.920 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:39.920 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:39.920 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51468888 partition_space_available_1k = 29621596 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1305708 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 14668 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46612 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14648772 free = 10210092 buffers = 678476 cached = 7127776 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.56 five_min_avg = 4.06 fifteen_min_avg = 4.04 cpu_share = 2.62 node_type = database-node >> >> 10/21/2024 02:11:39.920 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:39.920 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:11:39.921 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:11:39.921 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:39.921 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:11:39.921 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:39.921 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:39.922 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:11:39.922 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:43.921 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:11:48.926 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:11:48.926 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:11:48.927 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:48.927 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:11:48.927 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:48.927 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:11:48.929 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:11:48.929 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:48.929 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:11:48.929 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:48.929 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:48.930 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:11:48.931 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:52.928 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:11:57.933 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:11:57.933 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:11:57.933 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:57.934 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:11:57.934 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:11:57.934 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:11:57.934 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:11:57.934 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:57.935 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:11:57.935 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:57.935 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:11:57.935 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:11:57.935 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:01.936 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:12:06.938 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:12:06.939 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:12:06.939 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:12:06.939 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:12:06.939 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:12:06.939 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:12:06.940 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:12:06.940 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:06.940 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:12:06.940 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:06.940 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:06.940 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:12:06.940 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:10.940 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:12:15.945 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:12:15.945 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:12:15.946 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:12:15.946 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:12:15.946 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:12:15.946 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:12:15.946 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:12:15.947 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:15.947 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:12:15.947 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:15.947 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:15.947 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:12:15.947 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:19.948 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:12:24.953 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:12:24.953 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:12:24.953 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:12:24.954 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:12:24.954 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:12:24.954 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:12:24.954 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:12:24.955 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:24.955 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:12:24.955 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:24.955 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:24.955 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:12:24.955 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:28.511 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:12:28.869 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:28.870 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51478560 disk_space_available_1k = 29611924 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:12:28.870 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:28.958 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:12:33.962 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:12:33.962 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:12:33.962 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:12:33.963 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:12:33.963 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:12:33.963 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:12:33.963 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:12:33.964 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:33.964 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:12:33.964 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:33.964 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:33.964 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:12:33.964 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:36.897 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51479844, partition_space_available_1k=29610640, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1310328, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14876, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46652, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14685592, free=10152004, buffers=678484, cached=7149036, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.0, five_min_avg=5.13, fifteen_min_avg=4.43, cpu_share=5.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:12:36.897 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:36.898 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51479844 partition_space_available_1k = 29610640 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1310328 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 14876 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46652 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14685592 free = 10152004 buffers = 678484 cached = 7149036 node_type = database-node >> system_cpu_usage = << one_min_avg = 6.0 five_min_avg = 5.13 fifteen_min_avg = 4.43 cpu_share = 5.0 node_type = database-node >> >> 10/21/2024 02:12:36.898 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:37.962 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:12:42.968 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:12:42.969 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:12:42.969 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:12:42.969 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:12:42.969 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:12:42.970 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:12:42.970 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:12:42.970 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:42.971 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:12:42.971 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:42.971 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:42.971 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:12:42.971 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:46.972 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:12:51.974 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:12:51.974 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:12:51.975 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:12:51.975 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:12:51.975 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:12:51.975 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:12:51.975 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:12:51.976 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:51.976 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:12:51.976 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:51.976 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:51.976 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:12:51.976 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:12:55.977 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:13:00.981 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:13:00.982 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:13:00.982 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:00.982 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:13:00.982 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:00.982 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:13:00.983 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:13:00.983 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:00.983 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:13:00.983 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:00.983 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:00.984 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:13:00.984 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:04.985 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:13:09.598 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:13:09.987 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:13:09.987 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:13:09.987 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:09.987 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:13:09.987 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:09.987 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:13:09.988 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:13:09.988 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:09.988 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:13:09.988 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:09.988 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:09.988 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:13:09.989 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:13.989 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:13:18.995 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:13:18.995 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:13:18.995 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:18.996 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:13:18.996 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:18.996 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:13:18.996 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:13:18.997 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:18.997 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:13:18.997 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:18.997 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:18.997 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:13:18.997 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:22.996 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:13:23.411 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:13:28.002 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:13:28.002 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:13:28.002 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:28.002 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:13:28.002 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:28.003 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:13:28.003 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:13:28.003 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:28.004 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:13:28.004 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:28.004 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:28.004 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:13:28.004 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:28.495 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:13:28.850 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:28.850 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51487660 disk_space_available_1k = 29602824 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:13:28.850 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:32.002 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:13:36.922 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51487848, partition_space_available_1k=29602636, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1317764, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15088, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46700, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14759900, free=10071696, buffers=678484, cached=7155036, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.06, five_min_avg=4.72, fifteen_min_avg=4.33, cpu_share=2.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:13:37.005 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:13:37.005 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:13:37.006 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:37.006 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:13:37.006 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:37.006 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:37.006 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51487848 partition_space_available_1k = 29602636 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1317764 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 15088 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46700 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14759900 free = 10071696 buffers = 678484 cached = 7155036 node_type = database-node >> system_cpu_usage = << one_min_avg = 4.06 five_min_avg = 4.72 fifteen_min_avg = 4.33 cpu_share = 2.88 node_type = database-node >> >> 10/21/2024 02:13:37.006 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:37.007 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:13:37.007 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:13:37.007 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:37.007 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:13:37.008 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:37.008 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:37.008 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:13:37.008 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:40.569 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:13:41.006 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:13:46.011 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:13:46.011 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:13:46.011 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:46.012 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:13:46.012 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:46.012 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:13:46.012 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:13:46.012 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:46.013 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:13:46.013 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:46.013 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:46.013 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:13:46.013 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:50.014 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:13:55.018 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:13:55.018 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:13:55.018 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:55.018 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:13:55.019 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:13:55.019 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:13:55.019 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:13:55.020 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:55.020 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:13:55.020 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:55.021 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:55.021 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:13:55.021 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:13:59.021 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:14:04.026 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:14:04.026 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:14:04.026 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:04.027 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:14:04.027 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:04.027 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:14:04.027 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:14:04.028 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:04.028 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:14:04.028 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:04.028 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:04.028 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:14:04.028 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:08.026 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:14:13.032 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:14:13.032 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:14:13.032 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:13.033 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:14:13.034 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:13.034 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:14:13.034 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:14:13.035 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:13.035 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:14:13.035 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:13.035 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:13.035 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:14:13.035 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:17.035 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:14:22.037 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:14:22.037 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:14:22.038 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:22.038 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:14:22.038 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:22.038 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:14:22.038 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:14:22.039 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:22.039 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:14:22.039 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:22.039 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:22.039 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:14:22.039 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:26.040 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:14:28.485 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:14:31.043 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:14:31.043 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:14:31.043 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:31.044 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:14:31.044 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:31.044 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:31.044 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51487388 disk_space_available_1k = 29603096 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:14:31.044 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:31.044 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:14:31.045 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:14:31.047 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:31.047 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:14:31.048 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:31.048 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:31.048 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:14:31.048 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:35.044 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:14:36.725 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51487928, partition_space_available_1k=29602556, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1308060, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15276, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46736, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14742440, free=10081076, buffers=678516, cached=7163084, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.69, five_min_avg=4.2, fifteen_min_avg=4.18, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:14:40.048 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:14:40.049 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:14:40.049 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:40.049 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:14:40.049 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:40.049 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:40.050 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51487928 partition_space_available_1k = 29602556 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1308060 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 15276 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46736 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14742440 free = 10081076 buffers = 678516 cached = 7163084 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.69 five_min_avg = 4.2 fifteen_min_avg = 4.18 cpu_share = 2.75 node_type = database-node >> >> 10/21/2024 02:14:40.050 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:40.050 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:14:40.051 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:14:40.051 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:40.051 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:14:40.051 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:40.051 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:40.052 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:14:40.052 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:44.050 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:14:49.055 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:14:49.056 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:14:49.056 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:49.056 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:14:49.056 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:49.056 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:14:49.057 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:14:49.057 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:49.057 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:14:49.057 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:49.057 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:49.058 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:14:49.058 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:53.056 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:14:58.062 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:14:58.062 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:14:58.062 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:58.062 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:14:58.062 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:14:58.063 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:14:58.063 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:14:58.063 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:58.064 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:14:58.064 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:58.064 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:14:58.064 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:14:58.064 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:02.065 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:15:07.070 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:15:07.070 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:15:07.070 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:15:07.071 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:15:07.071 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:15:07.071 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:15:07.071 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:15:07.071 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:07.072 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:15:07.072 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:07.072 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:07.072 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:15:07.072 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:11.073 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:15:16.078 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:15:16.079 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:15:16.079 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:15:16.079 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:15:16.079 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:15:16.079 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:15:16.080 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:15:16.080 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:16.080 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:15:16.080 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:16.080 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:16.080 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:15:16.080 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:20.082 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:15:25.085 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:15:25.086 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:15:25.086 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:15:25.086 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:15:25.086 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:15:25.087 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:15:25.087 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:15:25.087 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:25.087 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:15:25.087 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:25.087 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:25.088 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:15:25.088 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:26.557 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:15:28.463 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:15:28.807 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:28.807 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51495520 disk_space_available_1k = 29594964 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:15:28.807 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:29.085 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:15:34.088 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:15:34.088 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:15:34.088 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:15:34.089 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:15:34.089 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:15:34.089 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:15:34.090 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:15:34.090 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:34.090 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:15:34.090 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:34.090 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:34.090 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:15:34.090 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:36.915 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51495676, partition_space_available_1k=29594808, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1313128, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15488, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46780, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14852028, free=9962384, buffers=678528, cached=7172176, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.84, five_min_avg=3.69, fifteen_min_avg=4.0, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:15:36.915 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:36.916 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51495676 partition_space_available_1k = 29594808 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1313128 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 15488 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46780 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14852028 free = 9962384 buffers = 678528 cached = 7172176 node_type = database-node >> system_cpu_usage = << one_min_avg = 1.84 five_min_avg = 3.69 fifteen_min_avg = 4.0 cpu_share = 2.75 node_type = database-node >> >> 10/21/2024 02:15:36.916 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:38.089 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:15:43.095 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:15:43.095 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:15:43.097 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:15:43.097 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:15:43.097 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:15:43.097 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:15:43.098 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:15:43.098 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:43.098 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:15:43.099 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:43.099 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:43.099 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:15:43.099 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:47.097 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:15:52.103 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:15:52.103 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:15:52.103 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:15:52.104 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:15:52.104 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:15:52.104 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:15:52.104 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:15:52.105 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:52.105 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:15:52.105 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:52.105 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:52.105 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:15:52.105 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:15:56.105 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:16:01.110 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:16:01.111 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:16:01.111 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:01.111 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:16:01.111 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:01.111 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:16:01.112 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:16:01.112 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:01.112 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:16:01.112 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:01.112 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:01.112 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:16:01.112 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:05.113 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:16:10.116 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:16:10.116 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:16:10.116 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:10.117 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:16:10.117 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:10.117 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:16:10.118 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:16:10.118 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:10.118 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:16:10.118 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:10.118 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:10.119 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:16:10.119 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:14.117 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:16:19.122 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:16:19.122 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:16:19.122 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:19.122 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:16:19.122 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:19.123 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:16:19.123 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:16:19.123 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:19.123 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:16:19.124 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:19.124 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:19.124 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:16:19.125 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:23.122 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:16:28.125 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:16:28.125 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:16:28.125 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:28.125 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:16:28.125 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:28.126 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:16:28.126 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:16:28.127 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:28.127 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:16:28.127 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:28.127 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:28.127 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:16:28.127 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:28.453 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:16:28.798 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:28.798 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51495520 disk_space_available_1k = 29594964 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:16:28.798 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:32.126 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:16:36.813 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51495512, partition_space_available_1k=29594972, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1309156, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15692, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46828, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14793284, free=10018764, buffers=678536, cached=7174532, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.24, five_min_avg=3.49, fifteen_min_avg=3.91, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:16:37.128 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:16:37.128 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:16:37.128 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:37.128 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:16:37.128 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:37.128 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:37.129 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51495512 partition_space_available_1k = 29594972 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1309156 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 15692 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46828 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14793284 free = 10018764 buffers = 678536 cached = 7174532 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.24 five_min_avg = 3.49 fifteen_min_avg = 3.91 cpu_share = 2.62 node_type = database-node >> >> 10/21/2024 02:16:37.129 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:37.129 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:16:37.129 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:16:37.130 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:37.130 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:16:37.130 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:37.130 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:37.130 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:16:37.130 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:41.129 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:16:46.134 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:16:46.135 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:16:46.135 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:46.135 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:16:46.135 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:46.135 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:16:46.136 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:16:46.136 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:46.136 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:16:46.136 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:46.136 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:46.136 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:16:46.136 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:50.138 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:16:55.141 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:16:55.141 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:16:55.141 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:55.141 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:16:55.141 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:16:55.142 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:16:55.142 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:16:55.142 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:55.142 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:16:55.142 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:55.143 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:55.143 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:16:55.143 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:16:59.143 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:17:04.148 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:17:04.148 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:17:04.148 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:04.148 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:17:04.148 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:04.149 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:17:04.149 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:17:04.149 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:04.149 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:17:04.149 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:04.150 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:04.150 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:17:04.150 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:08.151 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:17:13.157 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:17:13.157 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:17:13.157 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:13.158 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:17:13.158 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:13.158 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:17:13.158 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:17:13.159 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:13.159 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:17:13.159 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:13.159 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:13.159 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:17:13.159 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:17.159 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:17:19.798 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:17:22.161 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:17:22.162 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:17:22.162 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:22.162 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:17:22.162 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:22.162 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:17:22.163 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:17:22.163 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:22.163 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:17:22.163 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:22.163 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:22.163 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:17:22.163 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:26.163 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:17:28.606 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:17:31.164 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:17:31.181 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:17:31.181 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:31.181 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:17:31.181 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:31.181 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:31.182 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51506372 disk_space_available_1k = 29584112 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:17:31.182 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:31.182 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:17:31.182 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:17:31.183 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:31.183 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:17:31.183 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:31.183 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:31.183 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:17:31.183 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:35.144 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:17:35.165 7fd1113d8948 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:17:39.508 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51508288, partition_space_available_1k=29582196, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1312044, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15892, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46864, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14950596, free=9825328, buffers=679500, cached=7209692, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.29, five_min_avg=3.96, fifteen_min_avg=4.04, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:17:40.168 7fd1113d8948 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:17:40.168 7fd1113d8948 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:17:40.168 7fd1113d8948 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:40.168 7fd1113d8948 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:17:40.168 7fd1113d8948 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:40.168 7fd1113d8948 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:40.169 7fd1113d8948 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51508288 partition_space_available_1k = 29582196 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1312044 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 15892 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46864 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14950596 free = 9825328 buffers = 679500 cached = 7209692 node_type = database-node >> system_cpu_usage = << one_min_avg = 5.29 five_min_avg = 3.96 fifteen_min_avg = 4.04 cpu_share = 4.75 node_type = database-node >> >> 10/21/2024 02:17:40.169 7fd1113d8948 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:40.169 7fd1113d8948 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:17:40.170 7fd1113d8948 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:17:40.170 7fd1113d8948 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:40.170 7fd1113d8948 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:17:40.170 7fd1113d8948 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:40.170 7fd1113d8948 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:40.170 7fd1113d8948 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:17:40.170 7fd1113d8948 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:44.169 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:17:49.174 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:17:49.174 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:17:49.174 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:49.175 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:17:49.175 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:49.175 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:17:49.175 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:17:49.176 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:49.176 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:17:49.176 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:49.176 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:49.176 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:17:49.176 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:52.442 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:17:53.175 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:17:58.180 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:17:58.180 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:17:58.180 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:58.180 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:17:58.180 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:17:58.180 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:17:58.181 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:17:58.181 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:58.182 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:17:58.182 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:58.182 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:17:58.182 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:17:58.182 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:02.182 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:18:07.185 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:18:07.185 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:18:07.185 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:18:07.186 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:18:07.186 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:18:07.186 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:18:07.187 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:18:07.187 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:07.187 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:18:07.187 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:07.187 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:07.187 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:18:07.188 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:11.186 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:18:16.191 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:18:16.191 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:18:16.191 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:18:16.192 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:18:16.192 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:18:16.192 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:18:16.192 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:18:16.192 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:16.193 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:18:16.193 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:16.193 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:16.193 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:18:16.193 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:20.194 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:18:25.196 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:18:25.196 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:18:25.197 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:18:25.197 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:18:25.197 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:18:25.197 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:18:25.197 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:18:25.198 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:25.198 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:18:25.198 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:25.198 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:25.198 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:18:25.198 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:28.540 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:18:28.927 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:28.927 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51509440 disk_space_available_1k = 29581044 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:18:28.927 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:29.197 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:18:34.198 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:18:34.199 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:18:34.199 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:18:34.200 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:18:34.200 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:18:34.200 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:18:34.201 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:18:34.201 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:34.202 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:18:34.202 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:34.202 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:34.202 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:18:34.202 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:37.020 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51510860, partition_space_available_1k=29579624, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1310972, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16088, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46912, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14847640, free=9924088, buffers=679500, cached=7213888, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.33, five_min_avg=3.64, fifteen_min_avg=3.92, cpu_share=3.38, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:18:37.020 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:37.020 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51510860 partition_space_available_1k = 29579624 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1310972 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 16088 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46912 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14847640 free = 9924088 buffers = 679500 cached = 7213888 node_type = database-node >> system_cpu_usage = << one_min_avg = 3.33 five_min_avg = 3.64 fifteen_min_avg = 3.92 cpu_share = 3.38 node_type = database-node >> >> 10/21/2024 02:18:37.021 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:38.199 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:18:43.204 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:18:43.204 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:18:43.204 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:18:43.205 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:18:43.205 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:18:43.205 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:18:43.205 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:18:43.206 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:43.206 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:18:43.206 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:43.206 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:43.206 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:18:43.206 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:47.208 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:18:52.213 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:18:52.213 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:18:52.213 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:18:52.214 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:18:52.214 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:18:52.214 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:18:52.215 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:18:52.215 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:52.215 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:18:52.215 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:52.215 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:52.215 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:18:52.216 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:18:56.213 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:19:01.219 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:19:01.219 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:19:01.219 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:01.220 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:19:01.220 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:01.220 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:19:01.220 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:19:01.220 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:01.221 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:19:01.221 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:01.221 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:01.221 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:19:01.221 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:05.222 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:19:10.224 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:19:10.225 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:19:10.225 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:10.225 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:19:10.225 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:10.225 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:19:10.226 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:19:10.226 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:10.226 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:19:10.226 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:10.226 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:10.227 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:19:10.227 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:14.225 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:19:19.231 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:19:19.231 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:19:19.232 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:19.232 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:19:19.232 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:19.232 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:19:19.233 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:19:19.233 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:19.233 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:19:19.233 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:19.233 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:19.234 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:19:19.234 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:23.234 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:19:28.238 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:19:28.238 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:19:28.238 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:28.238 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:19:28.239 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:28.239 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:19:28.239 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:19:28.240 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:28.240 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:19:28.240 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:28.241 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:28.241 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:19:28.241 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:28.470 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:19:28.828 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:28.829 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51514180 disk_space_available_1k = 29576304 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:19:28.829 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:32.239 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:19:36.815 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51514336, partition_space_available_1k=29576148, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1310420, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16300, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=46960, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14820316, free=9943252, buffers=679616, cached=7221932, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.4, five_min_avg=3.34, fifteen_min_avg=3.8, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:19:37.241 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:19:37.242 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:19:37.242 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:37.242 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:19:37.242 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:37.242 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:37.243 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51514336 partition_space_available_1k = 29576148 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1310420 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 16300 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 46960 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14820316 free = 9943252 buffers = 679616 cached = 7221932 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.4 five_min_avg = 3.34 fifteen_min_avg = 3.8 cpu_share = 2.62 node_type = database-node >> >> 10/21/2024 02:19:37.243 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:37.243 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:19:37.243 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:19:37.243 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:37.244 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:19:37.244 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:37.244 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:37.244 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:19:37.244 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:41.242 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:19:46.249 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:19:46.249 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:19:46.249 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:46.249 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:19:46.250 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:46.250 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:19:46.250 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:19:46.250 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:46.251 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:19:46.251 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:46.251 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:46.251 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:19:46.251 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:50.251 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:19:55.254 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:19:55.254 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:19:55.254 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:55.255 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:19:55.255 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:19:55.255 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:19:55.255 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:19:55.255 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:55.256 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:19:55.256 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:55.256 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:55.256 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:19:55.256 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:19:59.257 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:20:04.262 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:20:04.263 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:20:04.263 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:04.263 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:20:04.263 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:04.263 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:20:04.264 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:20:04.264 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:04.264 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:20:04.264 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:04.264 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:04.264 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:20:04.264 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:08.266 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:20:13.269 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:20:13.269 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:20:13.269 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:13.269 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:20:13.270 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:13.270 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:20:13.270 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:20:13.270 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:13.270 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:20:13.271 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:13.271 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:13.271 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:20:13.271 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:17.272 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:20:22.277 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:20:22.277 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:20:22.278 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:22.278 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:20:22.278 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:22.278 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:20:22.279 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:20:22.279 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:22.279 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:20:22.279 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:22.279 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:22.279 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:20:22.279 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:26.278 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:20:28.500 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:20:31.281 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:20:31.281 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:20:31.281 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:31.282 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:20:31.282 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:31.282 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:31.282 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51517708 disk_space_available_1k = 29572776 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:20:31.282 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:31.283 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:20:31.283 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:20:31.283 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:31.284 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:20:31.284 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:31.284 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:31.284 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:20:31.284 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:35.282 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:20:36.787 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51517900, partition_space_available_1k=29572584, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1311940, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16472, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47004, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14817540, free=9941332, buffers=679620, cached=7226624, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.98, five_min_avg=3.06, fifteen_min_avg=3.67, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:20:40.287 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:20:40.288 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:20:40.288 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:40.288 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:20:40.288 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:40.288 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:40.289 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51517900 partition_space_available_1k = 29572584 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1311940 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 16472 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47004 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14817540 free = 9941332 buffers = 679620 cached = 7226624 node_type = database-node >> system_cpu_usage = << one_min_avg = 1.98 five_min_avg = 3.06 fifteen_min_avg = 3.67 cpu_share = 2.62 node_type = database-node >> >> 10/21/2024 02:20:40.289 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:40.289 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:20:40.290 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:20:40.290 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:40.290 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:20:40.290 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:40.290 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:40.290 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:20:40.290 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:44.291 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:20:49.297 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:20:49.297 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:20:49.298 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:49.298 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:20:49.298 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:49.298 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:20:49.299 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:20:49.299 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:49.299 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:20:49.299 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:49.299 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:49.299 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:20:49.299 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:53.300 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:20:58.304 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:20:58.304 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:20:58.304 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:58.304 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:20:58.305 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:20:58.305 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:20:58.305 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:20:58.306 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:58.306 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:20:58.306 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:58.306 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:20:58.306 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:20:58.307 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:02.305 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:21:07.310 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:21:07.310 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:21:07.311 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:21:07.311 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:21:07.311 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:21:07.311 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:21:07.312 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:21:07.312 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:07.312 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:21:07.312 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:07.312 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:07.313 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:21:07.313 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:11.311 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:21:14.785 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:21:16.313 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:21:16.314 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:21:16.314 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:21:16.314 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:21:16.315 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:21:16.315 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:21:16.316 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:21:16.317 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:16.317 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:21:16.317 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:16.317 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:16.317 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:21:16.318 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:20.317 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:21:25.323 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:21:25.323 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:21:25.323 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:21:25.323 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:21:25.324 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:21:25.324 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:21:25.324 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:21:25.324 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:25.325 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:21:25.325 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:25.325 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:25.325 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:21:25.325 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:28.485 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:21:28.839 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:28.840 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51521824 disk_space_available_1k = 29568660 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:21:28.840 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:29.323 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:21:34.325 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:21:34.325 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:21:34.325 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:21:34.326 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:21:34.326 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:21:34.326 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:21:34.327 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:21:34.327 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:34.327 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:21:34.327 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:34.327 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:34.327 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:21:34.327 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:36.858 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51521988, partition_space_available_1k=29568496, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1316724, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16688, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47044, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14822124, free=9931552, buffers=679684, cached=7231756, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.47, five_min_avg=3.01, fifteen_min_avg=3.62, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:21:36.858 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:36.859 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51521988 partition_space_available_1k = 29568496 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1316724 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 16688 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47044 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14822124 free = 9931552 buffers = 679684 cached = 7231756 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.47 five_min_avg = 3.01 fifteen_min_avg = 3.62 cpu_share = 2.62 node_type = database-node >> >> 10/21/2024 02:21:36.859 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:38.325 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:21:43.328 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:21:43.328 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:21:43.328 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:21:43.328 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:21:43.328 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:21:43.329 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:21:43.329 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:21:43.329 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:43.329 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:21:43.329 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:43.329 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:43.330 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:21:43.330 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:47.331 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:21:52.334 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:21:52.334 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:21:52.335 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:21:52.335 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:21:52.335 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:21:52.335 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:21:52.335 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:21:52.336 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:52.336 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:21:52.336 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:52.336 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:52.336 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:21:52.336 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:21:56.337 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:22:01.343 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:22:01.343 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:22:01.343 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:01.344 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:22:01.344 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:01.344 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:22:01.344 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:22:01.344 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:01.345 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:22:01.345 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:01.345 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:01.345 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:22:01.345 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:05.346 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:22:10.349 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:22:10.349 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:22:10.349 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:10.349 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:22:10.350 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:10.350 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:22:10.350 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:22:10.350 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:10.350 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:22:10.351 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:10.351 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:10.351 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:22:10.351 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:14.350 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:22:19.355 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:22:19.355 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:22:19.355 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:19.355 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:22:19.356 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:19.356 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:22:19.356 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:22:19.356 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:19.356 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:22:19.357 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:19.357 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:19.357 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:22:19.357 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:23.358 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:22:28.363 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:22:28.363 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:22:28.363 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:28.364 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:22:28.365 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:28.365 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:22:28.365 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:22:28.366 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:28.366 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:22:28.366 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:28.366 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:28.366 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:22:28.366 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:28.522 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:22:28.879 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:28.879 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51525728 disk_space_available_1k = 29564756 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:22:28.879 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:32.364 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:22:36.848 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51525896, partition_space_available_1k=29564588, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1315004, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16884, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47092, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14822480, free=9926396, buffers=679820, cached=7236420, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.48, five_min_avg=2.93, fifteen_min_avg=3.55, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:22:37.367 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:22:37.367 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:22:37.367 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:37.368 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:22:37.368 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:37.368 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:37.368 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51525896 partition_space_available_1k = 29564588 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1315004 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 16884 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47092 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14822480 free = 9926396 buffers = 679820 cached = 7236420 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.48 five_min_avg = 2.93 fifteen_min_avg = 3.55 cpu_share = 2.62 node_type = database-node >> >> 10/21/2024 02:22:37.368 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:37.368 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:22:37.369 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:22:37.369 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:37.369 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:22:37.369 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:37.369 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:37.370 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:22:37.370 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:41.369 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:22:46.373 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:22:46.373 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:22:46.374 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:46.374 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:22:46.374 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:46.374 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:22:46.375 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:22:46.375 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:46.375 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:22:46.375 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:46.375 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:46.376 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:22:46.376 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:50.376 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:22:51.845 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:22:55.381 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:22:55.381 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:22:55.381 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:55.382 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:22:55.382 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:22:55.382 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:22:55.382 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:22:55.382 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:55.383 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:22:55.383 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:55.383 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:55.383 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:22:55.383 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:22:59.383 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:23:04.389 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:23:04.389 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:23:04.389 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:04.390 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:23:04.390 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:04.390 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:23:04.391 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:23:04.391 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:04.391 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:23:04.391 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:04.391 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:04.391 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:23:04.391 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:05.574 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:23:08.391 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:23:13.393 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:23:13.393 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:23:13.393 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:13.394 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:23:13.394 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:13.394 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:23:13.395 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:23:13.395 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:13.395 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:23:13.395 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:13.395 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:13.395 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:23:13.395 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:17.395 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:23:22.401 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:23:22.401 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:23:22.401 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:22.402 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:23:22.402 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:22.402 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:23:22.402 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:23:22.403 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:22.403 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:23:22.403 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:22.403 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:22.403 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:23:22.403 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:26.403 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:23:28.602 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:23:29.423 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:23:31.405 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:23:31.406 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:23:31.406 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:31.406 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:23:31.406 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:31.406 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:31.407 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51537988 disk_space_available_1k = 29552496 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:23:31.407 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:31.407 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:23:31.407 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:23:31.409 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:31.409 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:23:31.410 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:31.410 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:31.410 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:23:31.410 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:35.407 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:23:37.726 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51539488, partition_space_available_1k=29550996, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1316068, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17092, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47128, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14942160, free=9780548, buffers=679820, cached=7262588, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.99, five_min_avg=3.69, fifteen_min_avg=3.76, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:23:40.410 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:23:40.410 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:23:40.411 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:40.411 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:23:40.411 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:40.411 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:40.411 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51539488 partition_space_available_1k = 29550996 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1316068 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 17092 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47128 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14942160 free = 9780548 buffers = 679820 cached = 7262588 node_type = database-node >> system_cpu_usage = << one_min_avg = 5.99 five_min_avg = 3.69 fifteen_min_avg = 3.76 cpu_share = 4.75 node_type = database-node >> >> 10/21/2024 02:23:40.411 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:40.411 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:23:40.412 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:23:40.412 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:40.412 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:23:40.412 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:40.413 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:40.413 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:23:40.413 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:44.411 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:23:49.416 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:23:49.416 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:23:49.416 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:49.417 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:23:49.417 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:49.417 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:23:49.417 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:23:49.417 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:49.418 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:23:49.418 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:49.418 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:49.418 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:23:49.418 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:53.417 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:23:58.420 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:23:58.421 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:23:58.421 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:58.421 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:23:58.421 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:23:58.421 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:23:58.422 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:23:58.422 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:58.422 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:23:58.422 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:58.422 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:23:58.423 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:23:58.423 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:02.423 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:24:07.428 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:24:07.428 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:24:07.428 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:24:07.428 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:24:07.428 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:24:07.429 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:24:07.429 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:24:07.429 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:07.429 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:24:07.429 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:07.430 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:07.430 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:24:07.430 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:11.431 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:24:16.434 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:24:16.435 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:24:16.435 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:24:16.435 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:24:16.435 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:24:16.435 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:24:16.436 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:24:16.436 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:16.436 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:24:16.436 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:16.436 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:16.437 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:24:16.437 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:20.437 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:24:25.442 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:24:25.442 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:24:25.442 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:24:25.442 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:24:25.442 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:24:25.442 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:24:25.443 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:24:25.443 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:25.443 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:24:25.443 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:25.443 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:25.443 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:24:25.444 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:28.473 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:24:28.879 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:28.879 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51540088 disk_space_available_1k = 29550396 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:24:28.879 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:29.442 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:24:34.445 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:24:34.445 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:24:34.445 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:24:34.446 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:24:34.446 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:24:34.446 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:24:34.447 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:24:34.447 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:34.447 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:24:34.447 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:34.447 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:34.447 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:24:34.447 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:36.909 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51540284, partition_space_available_1k=29550200, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1314792, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17304, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47172, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14858452, free=9862884, buffers=679836, cached=7263944, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.07, five_min_avg=3.29, fifteen_min_avg=3.62, cpu_share=2.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:24:36.910 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:36.910 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51540284 partition_space_available_1k = 29550200 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1314792 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 17304 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47172 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14858452 free = 9862884 buffers = 679836 cached = 7263944 node_type = database-node >> system_cpu_usage = << one_min_avg = 3.07 five_min_avg = 3.29 fifteen_min_avg = 3.62 cpu_share = 2.88 node_type = database-node >> >> 10/21/2024 02:24:36.910 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:38.446 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:24:43.448 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:24:43.449 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:24:43.449 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:24:43.449 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:24:43.449 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:24:43.449 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:24:43.450 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:24:43.450 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:43.450 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:24:43.450 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:43.450 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:43.450 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:24:43.451 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:47.452 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:24:52.457 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:24:52.457 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:24:52.457 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:24:52.458 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:24:52.458 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:24:52.458 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:24:52.458 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:24:52.459 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:52.459 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:24:52.459 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:52.459 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:52.459 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:24:52.459 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:24:56.458 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:25:01.462 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:25:01.462 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:25:01.462 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:01.463 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:25:01.463 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:01.463 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:25:01.463 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:25:01.464 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:01.464 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:25:01.464 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:01.464 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:01.464 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:25:01.464 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:05.465 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:25:10.471 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:25:10.471 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:25:10.472 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:10.472 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:25:10.472 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:10.472 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:25:10.473 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:25:10.473 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:10.473 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:25:10.473 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:10.473 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:10.473 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:25:10.473 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:14.471 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:25:19.477 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:25:19.477 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:25:19.477 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:19.478 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:25:19.478 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:19.478 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:25:19.479 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:25:19.479 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:19.479 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:25:19.479 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:19.479 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:19.479 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:25:19.480 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:23.478 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:25:28.469 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:25:28.484 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:25:28.484 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:25:28.484 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:28.485 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:25:28.485 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:28.485 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:25:28.485 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:25:28.485 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:28.486 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:25:28.486 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:28.486 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:28.486 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:25:28.486 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:28.872 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:28.873 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51543684 disk_space_available_1k = 29546800 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:25:28.873 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:32.485 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:25:36.850 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51543912, partition_space_available_1k=29546572, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1314740, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17504, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47504, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14853420, free=9857188, buffers=679848, cached=7274660, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.48, five_min_avg=3.35, fifteen_min_avg=3.62, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:25:37.487 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:25:37.488 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:25:37.488 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:37.488 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:25:37.488 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:37.488 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:37.488 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51543912 partition_space_available_1k = 29546572 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1314740 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 17504 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47504 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14853420 free = 9857188 buffers = 679848 cached = 7274660 node_type = database-node >> system_cpu_usage = << one_min_avg = 3.48 five_min_avg = 3.35 fifteen_min_avg = 3.62 cpu_share = 2.62 node_type = database-node >> >> 10/21/2024 02:25:37.489 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:37.489 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:25:37.489 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:25:37.489 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:37.489 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:25:37.490 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:37.490 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:37.490 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:25:37.490 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:41.490 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:25:46.494 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:25:46.494 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:25:46.494 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:46.494 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:25:46.495 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:46.495 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:25:46.495 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:25:46.495 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:46.495 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:25:46.496 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:46.496 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:46.496 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:25:46.496 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:50.497 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:25:55.502 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:25:55.502 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:25:55.502 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:55.502 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:25:55.502 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:25:55.502 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:25:55.503 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:25:55.503 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:55.503 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:25:55.503 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:55.503 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:55.504 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:25:55.504 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:25:59.503 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:26:04.508 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:26:04.509 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:26:04.509 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:04.509 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:26:04.509 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:04.510 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:26:04.510 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:26:04.510 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:04.511 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:26:04.511 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:04.511 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:04.511 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:26:04.511 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:08.509 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:26:13.514 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:26:13.514 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:26:13.514 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:13.515 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:26:13.515 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:13.515 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:26:13.516 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:26:13.516 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:13.516 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:26:13.516 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:13.516 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:13.516 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:26:13.517 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:17.517 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:26:22.523 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:26:22.523 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:26:22.523 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:22.523 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:26:22.523 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:22.524 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:26:22.524 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:26:22.524 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:22.525 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:26:22.525 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:22.525 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:22.525 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:26:22.525 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:26.526 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:26:28.491 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:26:31.528 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:26:31.529 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:26:31.529 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:31.529 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:26:31.529 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:31.529 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:31.529 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51547996 disk_space_available_1k = 29542488 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:26:31.529 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:31.529 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:26:31.530 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:26:31.530 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:31.530 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:26:31.530 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:31.530 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:31.530 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:26:31.531 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:35.529 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:26:36.885 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51548192, partition_space_available_1k=29542292, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1313456, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17700, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47552, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14856408, free=9849356, buffers=679848, cached=7279504, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.97, five_min_avg=2.97, fifteen_min_avg=3.47, cpu_share=2.5, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:26:40.534 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:26:40.534 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:26:40.534 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:40.534 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:26:40.534 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:40.535 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:40.535 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51548192 partition_space_available_1k = 29542292 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1313456 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 17700 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47552 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14856408 free = 9849356 buffers = 679848 cached = 7279504 node_type = database-node >> system_cpu_usage = << one_min_avg = 1.97 five_min_avg = 2.97 fifteen_min_avg = 3.47 cpu_share = 2.5 node_type = database-node >> >> 10/21/2024 02:26:40.535 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:40.535 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:26:40.535 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:26:40.536 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:40.536 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:26:40.536 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:40.536 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:40.536 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:26:40.536 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:44.534 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:26:49.540 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:26:49.540 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:26:49.540 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:49.541 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:26:49.541 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:49.541 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:26:49.541 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:26:49.542 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:49.542 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:26:49.542 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:49.542 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:49.542 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:26:49.542 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:53.543 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:26:58.549 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:26:58.549 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:26:58.549 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:58.550 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:26:58.550 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:26:58.550 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:26:58.550 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:26:58.550 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:58.551 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:26:58.551 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:58.551 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:26:58.551 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:26:58.551 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:02.549 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:27:07.543 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:27:07.551 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:27:07.551 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:27:07.551 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:27:07.551 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:27:07.551 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:27:07.551 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:27:07.552 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:27:07.552 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:07.552 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:27:07.552 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:07.552 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:07.552 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:27:07.552 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:11.554 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:27:16.556 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:27:16.557 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:27:16.557 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:27:16.557 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:27:16.557 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:27:16.557 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:27:16.558 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:27:16.558 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:16.558 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:27:16.558 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:16.558 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:16.558 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:27:16.558 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:20.559 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:27:25.563 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:27:25.563 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:27:25.563 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:27:25.563 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:27:25.564 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:27:25.564 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:27:25.565 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:27:25.565 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:25.565 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:27:25.565 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:25.565 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:25.565 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:27:25.566 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:28.481 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:27:28.849 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:28.849 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51547168 disk_space_available_1k = 29543316 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:27:28.850 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:29.567 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:27:34.572 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:27:34.573 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:27:34.573 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:27:34.573 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:27:34.574 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:27:34.574 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:27:34.574 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:27:34.575 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:34.575 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:27:34.575 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:34.575 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:34.575 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:27:34.575 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:36.857 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51547340, partition_space_available_1k=29543144, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1313876, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17856, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47584, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14846648, free=9859072, buffers=679852, cached=7279544, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.8, five_min_avg=2.76, fifteen_min_avg=3.37, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:27:36.858 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:36.858 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51547340 partition_space_available_1k = 29543144 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1313876 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 17856 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47584 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14846648 free = 9859072 buffers = 679852 cached = 7279544 node_type = database-node >> system_cpu_usage = << one_min_avg = 1.8 five_min_avg = 2.76 fifteen_min_avg = 3.37 cpu_share = 2.62 node_type = database-node >> >> 10/21/2024 02:27:36.858 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:38.573 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:27:43.578 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:27:43.578 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:27:43.578 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:27:43.578 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:27:43.578 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:27:43.578 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:27:43.579 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:27:43.579 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:43.579 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:27:43.579 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:43.579 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:43.580 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:27:43.580 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:45.263 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:27:47.579 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:27:52.584 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:27:52.584 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:27:52.584 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:27:52.584 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:27:52.585 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:27:52.585 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:27:52.586 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:27:52.586 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:52.586 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:27:52.586 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:52.586 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:52.587 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:27:52.587 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:27:56.586 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:27:59.033 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:28:01.592 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:28:01.592 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:28:01.592 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:28:01.592 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:28:01.592 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:28:01.593 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:28:01.593 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:28:01.593 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:01.593 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:28:01.594 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:01.594 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:01.594 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:28:01.594 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:05.595 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:28:10.600 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:28:10.600 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:28:10.600 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:28:10.601 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:28:10.601 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:28:10.601 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:28:10.602 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:28:10.602 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:10.602 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:28:10.602 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:10.602 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:10.603 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:28:10.603 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:14.602 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:28:17.790 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:28:19.605 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:28:19.605 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:28:19.605 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:28:19.606 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:28:19.606 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:28:19.607 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:28:19.607 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:28:19.607 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:19.608 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:28:19.608 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:19.608 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:19.608 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:28:19.608 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:23.606 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:28:28.549 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:28:28.609 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:28:28.635 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:28:28.636 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:28:28.636 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:28:28.636 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:28:28.636 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:28:28.637 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:28:28.637 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:28.637 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:28:28.637 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:28.638 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:28.638 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:28:28.638 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:29.269 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:29.269 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51557072 disk_space_available_1k = 29533412 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:28:29.270 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:32.610 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:28:37.613 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:28:37.613 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:28:37.614 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:28:37.614 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:28:37.614 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:28:37.614 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:28:37.615 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:28:37.615 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:37.615 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:28:37.615 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:37.616 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:37.616 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:28:37.616 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:37.745 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51557352, partition_space_available_1k=29533132, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1314704, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17884, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47632, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14922036, free=9762820, buffers=679852, cached=7300408, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.45, five_min_avg=3.96, fifteen_min_avg=3.74, cpu_share=5.0, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:28:37.745 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:37.746 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51557352 partition_space_available_1k = 29533132 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1314704 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 17884 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47632 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14922036 free = 9762820 buffers = 679852 cached = 7300408 node_type = database-node >> system_cpu_usage = << one_min_avg = 6.45 five_min_avg = 3.96 fifteen_min_avg = 3.74 cpu_share = 5.0 node_type = database-node >> >> 10/21/2024 02:28:37.746 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:41.616 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:28:46.618 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:28:46.618 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:28:46.618 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:28:46.619 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:28:46.619 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:28:46.619 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:28:46.619 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:28:46.619 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:46.620 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:28:46.620 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:46.620 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:46.620 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:28:46.620 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:50.621 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:28:55.625 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:28:55.626 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:28:55.626 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:28:55.626 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:28:55.626 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:28:55.626 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:28:55.627 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:28:55.627 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:55.627 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:28:55.627 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:55.627 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:55.627 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:28:55.627 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:28:59.629 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:29:04.633 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:29:04.633 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:29:04.633 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:04.634 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:29:04.634 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:04.634 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:29:04.634 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:29:04.634 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:04.635 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:29:04.635 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:04.635 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:04.635 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:29:04.635 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:08.636 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:29:13.641 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:29:13.642 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:29:13.642 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:13.642 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:29:13.642 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:13.642 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:29:13.643 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:29:13.643 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:13.643 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:29:13.643 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:13.643 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:13.644 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:29:13.644 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:17.642 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:29:22.647 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:29:22.648 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:29:22.648 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:22.648 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:29:22.648 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:22.648 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:29:22.649 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:29:22.649 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:22.649 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:29:22.649 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:22.649 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:22.649 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:29:22.649 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:26.650 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:29:28.480 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:29:31.652 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:29:31.652 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:29:31.652 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:31.652 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:29:31.652 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:31.653 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:31.653 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51562224 disk_space_available_1k = 29528260 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:29:31.653 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:31.653 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:29:31.653 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:29:31.654 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:31.654 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:29:31.655 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:31.655 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:31.655 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:29:31.655 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:35.653 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:29:36.857 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51562436, partition_space_available_1k=29528048, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1313780, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17920, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47676, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14880204, free=9799940, buffers=679912, cached=7305060, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.45, five_min_avg=3.56, fifteen_min_avg=3.62, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:29:40.657 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:29:40.658 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:29:40.658 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:40.658 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:29:40.658 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:40.659 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:40.660 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51562436 partition_space_available_1k = 29528048 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1313780 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 17920 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47676 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14880204 free = 9799940 buffers = 679912 cached = 7305060 node_type = database-node >> system_cpu_usage = << one_min_avg = 3.45 five_min_avg = 3.56 fifteen_min_avg = 3.62 cpu_share = 2.62 node_type = database-node >> >> 10/21/2024 02:29:40.660 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:40.660 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:29:40.661 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:29:40.661 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:40.661 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:29:40.662 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:40.663 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:40.663 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:29:40.663 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:44.658 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:29:49.662 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:29:49.662 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:29:49.662 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:49.662 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:29:49.663 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:49.663 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:29:49.663 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:29:49.664 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:49.664 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:29:49.664 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:49.664 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:49.664 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:29:49.664 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:53.665 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:29:58.670 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:29:58.671 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:29:58.671 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:58.671 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:29:58.671 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:29:58.671 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:29:58.672 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:29:58.672 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:58.672 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:29:58.672 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:58.672 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:29:58.672 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:29:58.673 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:02.671 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:30:07.676 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:30:07.677 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:30:07.677 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:30:07.677 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:30:07.677 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:30:07.677 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:30:07.678 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:30:07.678 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:07.679 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:30:07.679 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:07.679 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:07.679 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:30:07.679 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:11.679 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:30:16.686 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:30:16.686 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:30:16.687 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:30:16.687 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:30:16.687 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:30:16.687 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:30:16.688 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:30:16.688 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:16.688 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:30:16.688 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:16.688 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:16.688 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:30:16.688 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:20.689 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:30:25.694 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:30:25.694 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:30:25.694 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:30:25.694 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:30:25.694 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:30:25.695 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:30:25.695 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:30:25.695 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:25.695 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:30:25.695 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:25.696 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:25.696 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:30:25.696 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:28.479 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:30:28.869 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:28.870 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51561096 disk_space_available_1k = 29529388 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:30:28.870 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:29.694 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:30:34.698 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:30:34.698 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:30:34.698 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:30:34.699 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:30:34.699 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:30:34.699 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:30:34.700 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:30:34.700 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:34.700 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:30:34.700 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:34.700 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:34.701 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:30:34.701 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:36.932 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51561276, partition_space_available_1k=29529208, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1313652, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17948, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47720, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14900840, free=9779040, buffers=679912, cached=7305324, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.95, five_min_avg=3.42, fifteen_min_avg=3.57, cpu_share=2.62, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:30:36.932 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:36.933 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51561276 partition_space_available_1k = 29529208 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1313652 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 17948 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47720 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14900840 free = 9779040 buffers = 679912 cached = 7305324 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.95 five_min_avg = 3.42 fifteen_min_avg = 3.57 cpu_share = 2.62 node_type = database-node >> >> 10/21/2024 02:30:36.933 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:38.699 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:30:43.705 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:30:43.705 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:30:43.705 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:30:43.705 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:30:43.706 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:30:43.706 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:30:43.706 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:30:43.707 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:43.707 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:30:43.707 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:43.707 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:43.707 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:30:43.707 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:47.706 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:30:52.711 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:30:52.712 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:30:52.712 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:30:52.712 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:30:52.712 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:30:52.712 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:30:52.713 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:30:52.713 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:52.713 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:30:52.714 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:52.714 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:52.714 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:30:52.714 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:30:56.714 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:31:01.719 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:31:01.720 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:31:01.720 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:01.720 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:31:01.720 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:01.720 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:31:01.721 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:31:01.721 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:01.721 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:31:01.721 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:01.721 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:01.722 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:31:01.722 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:05.722 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:31:10.727 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:31:10.727 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:31:10.727 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:10.727 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:31:10.728 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:10.728 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:31:10.728 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:31:10.728 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:10.728 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:31:10.729 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:10.729 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:10.729 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:31:10.729 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:14.730 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:31:19.733 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:31:19.733 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:31:19.733 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:19.733 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:31:19.734 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:19.734 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:31:19.734 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:31:19.735 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:19.735 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:31:19.735 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:19.735 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:19.735 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:31:19.735 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:23.735 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:31:28.494 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:31:28.740 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:31:28.741 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:31:28.741 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:28.741 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:31:28.741 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:28.741 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:31:28.742 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:31:28.742 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:28.742 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:31:28.742 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:28.742 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:28.742 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:31:28.742 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:28.847 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:28.847 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51565416 disk_space_available_1k = 29525068 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:31:28.847 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:32.741 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:31:36.861 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51565588, partition_space_available_1k=29524896, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1313796, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17992, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47760, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14936256, free=9738660, buffers=679916, cached=7310284, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.0, five_min_avg=3.37, fifteen_min_avg=3.54, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:31:37.743 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:31:37.744 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:31:37.744 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:37.744 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:31:37.744 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:37.744 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:37.744 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51565588 partition_space_available_1k = 29524896 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1313796 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 17992 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47760 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14936256 free = 9738660 buffers = 679916 cached = 7310284 node_type = database-node >> system_cpu_usage = << one_min_avg = 3.0 five_min_avg = 3.37 fifteen_min_avg = 3.54 cpu_share = 2.75 node_type = database-node >> >> 10/21/2024 02:31:37.744 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:37.745 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:31:37.745 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:31:37.745 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:37.745 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:31:37.746 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:37.746 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:37.746 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:31:37.746 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:41.747 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:31:44.735 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:31:46.750 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:31:46.751 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:31:46.751 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:46.751 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:31:46.751 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:46.751 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:31:46.752 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:31:46.752 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:46.752 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:31:46.752 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:46.752 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:46.752 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:31:46.753 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:50.753 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:31:55.758 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:31:55.759 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:31:55.759 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:55.759 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:31:55.759 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:31:55.759 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:31:55.760 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:31:55.760 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:55.760 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:31:55.760 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:55.760 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:55.761 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:31:55.761 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:31:59.761 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:32:04.763 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:32:04.764 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:32:04.764 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:04.764 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:32:04.764 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:04.764 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:32:04.765 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:32:04.765 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:04.765 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:32:04.765 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:04.765 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:04.766 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:32:04.766 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:08.765 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:32:13.770 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:32:13.770 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:32:13.770 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:13.770 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:32:13.770 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:13.771 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:32:13.771 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:32:13.771 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:13.771 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:32:13.771 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:13.772 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:13.772 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:32:13.772 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:17.773 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:32:22.781 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:32:22.781 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:32:22.781 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:22.782 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:32:22.782 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:22.782 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:32:22.782 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:32:22.782 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:22.783 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:32:22.783 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:22.783 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:22.783 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:32:22.783 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:26.374 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:32:26.782 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:32:28.551 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:32:31.784 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:32:31.784 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:32:31.784 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:31.785 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:32:31.785 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:31.785 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:31.785 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51568748 disk_space_available_1k = 29521736 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:32:31.785 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:31.785 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:32:31.786 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:32:31.786 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:31.786 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:32:31.786 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:31.786 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:31.786 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:32:31.786 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:35.784 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:32:36.925 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51568936, partition_space_available_1k=29521548, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1314112, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18020, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47812, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14989032, free=9682228, buffers=679916, cached=7313940, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.81, five_min_avg=3.24, fifteen_min_avg=3.48, cpu_share=2.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:32:40.341 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:32:40.787 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:32:40.787 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:32:40.787 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:40.788 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:32:40.788 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:40.788 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:40.788 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51568936 partition_space_available_1k = 29521548 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1314112 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 18020 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47812 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14989032 free = 9682228 buffers = 679916 cached = 7313940 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.81 five_min_avg = 3.24 fifteen_min_avg = 3.48 cpu_share = 2.88 node_type = database-node >> >> 10/21/2024 02:32:40.788 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:40.788 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:32:40.789 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:32:40.789 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:40.789 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:32:40.789 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:40.789 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:40.789 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:32:40.789 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:44.790 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:32:49.793 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:32:49.793 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:32:49.793 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:49.793 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:32:49.794 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:49.794 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:32:49.794 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:32:49.794 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:49.794 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:32:49.795 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:49.795 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:49.795 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:32:49.795 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:53.795 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:32:57.341 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:32:58.797 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:32:58.797 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:32:58.797 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:58.798 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:32:58.798 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:32:58.798 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:32:58.798 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:32:58.798 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:58.799 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:32:58.799 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:58.799 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:32:58.799 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:32:58.799 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:02.799 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:33:07.802 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:33:07.802 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:33:07.802 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:07.802 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:33:07.803 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:07.803 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:33:07.803 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:33:07.804 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:07.804 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:33:07.804 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:07.804 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:07.804 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:33:07.804 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:11.805 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:33:16.809 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:33:16.810 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:33:16.810 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:16.810 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:33:16.810 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:16.810 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:33:16.811 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:33:16.811 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:16.811 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:33:16.811 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:16.811 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:16.812 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:33:16.812 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:20.811 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:33:25.816 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:33:25.816 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:33:25.816 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:25.816 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:33:25.816 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:25.817 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:33:25.817 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:33:25.817 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:25.817 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:33:25.818 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:25.818 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:25.818 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:33:25.818 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:29.395 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:33:29.830 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:33:34.832 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:33:34.832 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:33:34.832 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:34.833 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:33:34.833 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:34.833 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:34.833 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51672988 disk_space_available_1k = 29417496 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:33:34.834 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:34.834 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:33:34.834 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:33:34.834 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:34.835 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:33:34.835 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:34.835 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:34.835 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:33:34.835 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:38.833 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:33:42.233 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51574504, partition_space_available_1k=29515980, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1314624, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18056, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47848, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15036032, free=9710116, buffers=679920, cached=7239048, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.45, five_min_avg=3.7, fifteen_min_avg=3.61, cpu_share=4.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:33:43.836 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:33:43.836 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:33:43.836 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:43.837 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:33:43.837 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:43.837 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:43.837 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51574504 partition_space_available_1k = 29515980 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1314624 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 18056 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47848 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 15036032 free = 9710116 buffers = 679920 cached = 7239048 node_type = database-node >> system_cpu_usage = << one_min_avg = 5.45 five_min_avg = 3.7 fifteen_min_avg = 3.61 cpu_share = 4.75 node_type = database-node >> >> 10/21/2024 02:33:43.837 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:43.837 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:33:43.838 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:33:43.838 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:43.838 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:33:43.838 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:43.838 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:43.839 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:33:43.839 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:47.839 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:33:52.844 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:33:52.844 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:33:52.845 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:52.845 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:33:52.845 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:33:52.845 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:33:52.846 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:33:52.846 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:52.846 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:33:52.846 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:52.846 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:52.846 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:33:52.846 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:33:56.847 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:34:01.852 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:34:01.853 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:34:01.853 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:01.853 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:34:01.853 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:01.853 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:34:01.854 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:34:01.854 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:01.854 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:34:01.854 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:01.854 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:01.854 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:34:01.854 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:05.852 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:34:10.857 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:34:10.858 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:34:10.858 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:10.858 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:34:10.858 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:10.858 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:34:10.859 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:34:10.859 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:10.859 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:34:10.859 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:10.859 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:10.859 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:34:10.860 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:14.861 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:34:19.864 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:34:19.864 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:34:19.864 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:19.865 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:34:19.865 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:19.865 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:34:19.867 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:34:19.867 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:19.868 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:34:19.868 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:19.868 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:19.868 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:34:19.869 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:23.866 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:34:28.541 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:34:28.871 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:34:28.872 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:34:28.872 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:28.872 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:34:28.872 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:28.872 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:34:28.873 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:34:28.873 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:28.873 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:34:28.873 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:28.873 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:28.873 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:34:28.873 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:28.935 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:28.936 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51574500 disk_space_available_1k = 29515984 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:34:28.936 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:32.872 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:34:36.965 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51574696, partition_space_available_1k=29515788, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1314984, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18088, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47896, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14966224, free=9779280, buffers=679920, cached=7239692, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.07, five_min_avg=3.66, fifteen_min_avg=3.61, cpu_share=2.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:34:37.875 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:34:37.875 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:34:37.875 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:37.876 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:34:37.876 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:37.876 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:37.876 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51574696 partition_space_available_1k = 29515788 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1314984 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 18088 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47896 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14966224 free = 9779280 buffers = 679920 cached = 7239692 node_type = database-node >> system_cpu_usage = << one_min_avg = 4.07 five_min_avg = 3.66 fifteen_min_avg = 3.61 cpu_share = 2.88 node_type = database-node >> >> 10/21/2024 02:34:37.876 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:37.876 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:34:37.877 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:34:37.877 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:37.877 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:34:37.877 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:37.877 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:37.878 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:34:37.878 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:41.877 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:34:46.881 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:34:46.881 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:34:46.881 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:46.881 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:34:46.882 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:46.882 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:34:46.882 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:34:46.882 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:46.882 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:34:46.883 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:46.883 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:46.883 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:34:46.883 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:50.882 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:34:55.885 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:34:55.885 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:34:55.885 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:55.886 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:34:55.886 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:34:55.886 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:34:55.886 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:34:55.886 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:55.887 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:34:55.887 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:55.887 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:55.887 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:34:55.887 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:34:59.886 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:35:04.888 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:35:04.888 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:35:04.889 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:04.889 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:35:04.889 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:04.889 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:35:04.890 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:35:04.890 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:04.890 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:35:04.890 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:04.890 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:04.890 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:35:04.891 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:08.889 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:35:13.893 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:35:13.893 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:35:13.893 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:13.893 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:35:13.893 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:13.894 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:35:13.894 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:35:13.894 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:13.894 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:35:13.894 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:13.895 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:13.895 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:35:13.895 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:17.896 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:35:22.900 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:35:22.900 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:35:22.900 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:22.900 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:35:22.901 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:22.901 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:35:22.902 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:35:22.902 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:22.902 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:35:22.902 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:22.902 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:22.902 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:35:22.903 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:26.903 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:35:28.513 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:35:31.906 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:35:31.906 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:35:31.907 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:31.907 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:35:31.907 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:31.907 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:31.907 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51578248 disk_space_available_1k = 29512236 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:35:31.907 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:31.907 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:35:31.908 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:35:31.908 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:31.909 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:35:31.909 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:31.909 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:31.909 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:35:31.909 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:35.906 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:35:36.772 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51578396, partition_space_available_1k=29512088, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1314384, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18120, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47936, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14956516, free=9783120, buffers=679920, cached=7245560, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.53, five_min_avg=3.31, fifteen_min_avg=3.49, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:35:40.911 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:35:40.911 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:35:40.911 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:40.911 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:35:40.911 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:40.912 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:40.912 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51578396 partition_space_available_1k = 29512088 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1314384 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 18120 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47936 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14956516 free = 9783120 buffers = 679920 cached = 7245560 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.53 five_min_avg = 3.31 fifteen_min_avg = 3.49 cpu_share = 2.75 node_type = database-node >> >> 10/21/2024 02:35:40.912 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:40.912 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:35:40.913 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:35:40.913 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:40.913 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:35:40.913 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:40.913 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:40.913 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:35:40.913 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:44.914 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:35:49.917 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:35:49.917 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:35:49.918 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:49.918 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:35:49.918 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:49.918 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:35:49.919 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:35:49.919 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:49.919 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:35:49.919 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:49.919 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:49.919 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:35:49.919 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:53.919 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:35:58.627 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:35:58.922 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:35:58.922 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:35:58.922 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:58.922 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:35:58.923 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:35:58.923 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:35:58.923 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:35:58.923 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:58.924 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:35:58.924 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:58.924 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:35:58.924 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:35:58.924 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:02.925 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:36:07.929 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:36:07.929 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:36:07.929 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:36:07.929 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:36:07.929 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:36:07.930 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:36:07.930 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:36:07.930 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:07.930 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:36:07.931 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:07.931 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:07.931 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:36:07.931 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:11.932 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:36:16.937 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:36:16.937 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:36:16.937 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:36:16.937 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:36:16.937 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:36:16.938 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:36:16.938 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:36:16.938 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:16.938 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:36:16.938 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:16.939 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:16.939 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:36:16.939 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:20.938 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:36:25.941 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:36:25.942 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:36:25.942 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:36:25.942 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:36:25.942 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:36:25.942 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:36:25.943 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:36:25.943 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:25.943 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:36:25.943 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:25.943 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:25.943 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:36:25.943 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:28.540 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:36:28.926 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:28.927 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51583040 disk_space_available_1k = 29507444 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:36:28.927 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:29.942 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:36:34.945 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:36:34.945 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:36:34.945 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:36:34.946 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:36:34.946 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:36:34.946 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:36:34.947 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:36:34.947 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:34.947 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:36:34.947 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:34.947 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:34.947 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:36:34.947 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:36.918 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51583188, partition_space_available_1k=29507296, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1314612, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18460, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=47984, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14959328, free=9774984, buffers=679924, cached=7250880, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.18, five_min_avg=3.08, fifteen_min_avg=3.4, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:36:36.919 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:36.919 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51583188 partition_space_available_1k = 29507296 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1314612 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 18460 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 47984 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14959328 free = 9774984 buffers = 679924 cached = 7250880 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.18 five_min_avg = 3.08 fifteen_min_avg = 3.4 cpu_share = 2.75 node_type = database-node >> >> 10/21/2024 02:36:36.919 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:38.946 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:36:43.949 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:36:43.949 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:36:43.949 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:36:43.949 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:36:43.949 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:36:43.950 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:36:43.950 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:36:43.950 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:43.951 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:36:43.951 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:43.951 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:43.951 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:36:43.951 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:47.952 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:36:52.954 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:36:52.954 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:36:52.955 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:36:52.955 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:36:52.955 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:36:52.955 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:36:52.956 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:36:52.956 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:52.956 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:36:52.956 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:52.956 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:52.956 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:36:52.957 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:36:56.956 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:37:01.961 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:37:01.961 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:37:01.961 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:01.961 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:37:01.961 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:01.962 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:37:01.962 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:37:01.962 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:01.962 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:37:01.962 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:01.963 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:01.963 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:37:01.963 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:05.961 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:37:10.967 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:37:10.967 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:37:10.967 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:10.968 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:37:10.968 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:10.968 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:37:10.969 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:37:10.969 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:10.969 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:37:10.969 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:10.969 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:10.969 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:37:10.969 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:14.970 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:37:19.975 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:37:19.976 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:37:19.976 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:19.976 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:37:19.976 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:19.977 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:37:19.977 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:37:19.978 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:19.978 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:37:19.978 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:19.978 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:19.978 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:37:19.978 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:23.978 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:37:28.513 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:37:28.985 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:37:28.985 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:37:28.985 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:28.985 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:37:28.986 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:28.986 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:28.986 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51582256 disk_space_available_1k = 29508228 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:37:28.986 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:28.986 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:37:28.987 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:37:28.987 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:28.987 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:37:28.987 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:28.987 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:28.987 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:37:28.987 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:32.992 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:37:37.014 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51582476, partition_space_available_1k=29508008, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1322660, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18472, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48028, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14947908, free=9786480, buffers=679924, cached=7250804, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.04, five_min_avg=2.88, fifteen_min_avg=3.31, cpu_share=2.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:37:37.995 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:37:37.995 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:37:37.995 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:37.996 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:37:37.996 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:37.996 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:37.996 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51582476 partition_space_available_1k = 29508008 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1322660 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 18472 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 48028 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14947908 free = 9786480 buffers = 679924 cached = 7250804 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.04 five_min_avg = 2.88 fifteen_min_avg = 3.31 cpu_share = 2.88 node_type = database-node >> >> 10/21/2024 02:37:37.997 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:37.997 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:37:37.997 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:37:37.998 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:37.998 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:37:37.998 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:37.998 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:37.999 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:37:37.999 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:41.997 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:37:47.001 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:37:47.001 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:37:47.001 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:47.002 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:37:47.002 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:47.002 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:37:47.003 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:37:47.003 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:47.003 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:37:47.003 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:47.004 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:47.004 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:37:47.004 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:51.003 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:37:56.009 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:37:56.009 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:37:56.009 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:56.009 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:37:56.009 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:37:56.009 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:37:56.010 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:37:56.010 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:56.010 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:37:56.010 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:56.011 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:56.011 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:37:56.011 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:37:59.845 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:38:00.010 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:38:05.015 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:38:05.015 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:38:05.016 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:05.016 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:38:05.016 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:05.016 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:38:05.016 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:38:05.017 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:05.017 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:38:05.017 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:05.017 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:05.017 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:38:05.017 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:09.017 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:38:13.862 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:38:14.019 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:38:14.019 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:38:14.019 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:14.020 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:38:14.020 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:14.020 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:38:14.020 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:38:14.021 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:14.021 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:38:14.021 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:14.021 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:14.021 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:38:14.021 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:18.022 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:38:23.024 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:38:23.024 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:38:23.024 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:23.025 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:38:23.025 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:23.025 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:38:23.025 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:38:23.025 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:23.026 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:38:23.026 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:23.026 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:23.026 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:38:23.026 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:27.026 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:38:28.512 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:38:31.218 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:38:32.027 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:38:32.028 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:38:32.028 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:32.028 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:38:32.028 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:32.028 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:32.029 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51586432 disk_space_available_1k = 29504052 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:38:32.029 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:32.029 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:38:32.029 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:38:32.030 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:32.030 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:38:32.030 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:32.030 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:32.030 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:38:32.030 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:36.028 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:38:37.120 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51590752, partition_space_available_1k=29499732, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1317844, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18492, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48068, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15034848, free=9688892, buffers=679924, cached=7261452, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.73, five_min_avg=2.89, fifteen_min_avg=3.28, cpu_share=3.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:38:41.033 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:38:41.033 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:38:41.033 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:41.033 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:38:41.034 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:41.034 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:41.034 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51590752 partition_space_available_1k = 29499732 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1317844 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 18492 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 48068 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 15034848 free = 9688892 buffers = 679924 cached = 7261452 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.73 five_min_avg = 2.89 fifteen_min_avg = 3.28 cpu_share = 3.25 node_type = database-node >> >> 10/21/2024 02:38:41.034 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:41.034 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:38:41.035 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:38:41.035 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:41.035 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:38:41.035 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:41.035 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:41.035 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:38:41.035 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:45.034 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:38:50.039 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:38:50.040 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:38:50.040 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:50.040 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:38:50.041 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:50.041 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:38:50.041 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:38:50.042 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:50.042 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:38:50.042 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:50.042 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:50.042 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:38:50.042 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:54.041 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:38:59.046 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:38:59.047 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:38:59.047 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:59.047 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:38:59.047 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:38:59.047 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:38:59.048 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:38:59.048 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:59.048 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:38:59.048 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:59.048 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:38:59.049 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:38:59.049 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:03.050 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:39:08.052 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:39:08.052 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:39:08.052 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:39:08.052 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:39:08.053 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:39:08.053 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:39:08.053 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:39:08.053 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:08.054 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:39:08.054 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:08.054 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:08.054 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:39:08.054 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:12.054 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:39:17.059 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:39:17.059 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:39:17.059 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:39:17.060 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:39:17.060 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:39:17.060 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:39:17.060 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:39:17.061 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:17.061 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:39:17.061 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:17.061 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:17.061 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:39:17.061 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:21.065 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:39:26.071 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:39:26.071 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:39:26.072 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:39:26.072 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:39:26.072 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:39:26.073 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:39:26.074 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:39:26.074 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:26.075 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:39:26.075 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:26.075 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:26.076 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:39:26.076 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:28.571 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:39:28.973 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:28.974 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51591264 disk_space_available_1k = 29499220 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:39:28.974 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:30.072 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:39:35.075 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:39:35.076 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:39:35.076 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:39:35.076 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:39:35.076 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:39:35.077 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:39:35.078 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:39:35.078 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:35.078 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:39:35.078 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:35.079 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:35.079 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:39:35.079 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:37.237 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51591568, partition_space_available_1k=29498916, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1315100, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18516, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48116, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14993504, free=9718804, buffers=679924, cached=7272884, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.1, five_min_avg=3.48, fifteen_min_avg=3.45, cpu_share=5.25, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:39:37.237 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:37.238 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51591568 partition_space_available_1k = 29498916 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1315100 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 18516 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 48116 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14993504 free = 9718804 buffers = 679924 cached = 7272884 node_type = database-node >> system_cpu_usage = << one_min_avg = 5.1 five_min_avg = 3.48 fifteen_min_avg = 3.45 cpu_share = 5.25 node_type = database-node >> >> 10/21/2024 02:39:37.238 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:39.076 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:39:44.082 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:39:44.082 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:39:44.082 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:39:44.082 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:39:44.083 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:39:44.083 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:39:44.083 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:39:44.083 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:44.083 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:39:44.084 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:44.084 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:44.084 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:39:44.084 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:48.085 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:39:53.088 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:39:53.088 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:39:53.089 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:39:53.089 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:39:53.089 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:39:53.089 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:39:53.090 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:39:53.090 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:53.090 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:39:53.090 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:53.090 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:53.090 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:39:53.090 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:39:57.089 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:40:02.094 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:40:02.095 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:40:02.095 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:02.096 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:40:02.096 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:02.096 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:40:02.097 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:40:02.097 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:02.097 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:40:02.097 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:02.098 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:02.098 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:40:02.098 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:06.097 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:40:11.100 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:40:11.101 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:40:11.101 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:11.101 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:40:11.101 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:11.101 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:40:11.102 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:40:11.103 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:11.104 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:40:11.104 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:11.104 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:11.104 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:40:11.104 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:15.101 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:40:20.105 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:40:20.105 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:40:20.106 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:20.106 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:40:20.106 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:20.106 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:40:20.107 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:40:20.107 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:20.107 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:40:20.107 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:20.108 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:20.108 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:40:20.108 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:24.105 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:40:25.415 7fd1113d8948 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:40:28.566 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:40:29.108 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:40:29.108 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:40:29.108 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:29.109 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:40:29.109 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:29.109 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:29.109 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51600284 disk_space_available_1k = 29490200 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:40:29.109 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:29.110 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:40:29.110 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:40:29.111 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:29.111 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:40:29.111 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:29.111 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:29.112 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:40:29.112 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:33.108 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:40:36.996 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51600532, partition_space_available_1k=29489952, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1318768, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18532, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48152, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=15016432, free=9667944, buffers=681916, cached=7298824, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.05, five_min_avg=3.18, fifteen_min_avg=3.35, cpu_share=2.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:40:38.111 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:40:38.112 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:40:38.112 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:38.112 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:40:38.112 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:38.112 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:38.112 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51600532 partition_space_available_1k = 29489952 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1318768 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 18532 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 48152 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 15016432 free = 9667944 buffers = 681916 cached = 7298824 node_type = database-node >> system_cpu_usage = << one_min_avg = 3.05 five_min_avg = 3.18 fifteen_min_avg = 3.35 cpu_share = 2.88 node_type = database-node >> >> 10/21/2024 02:40:38.113 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:38.113 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:40:38.113 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:40:38.113 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:38.114 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:40:38.114 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:38.114 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:38.114 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:40:38.114 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:42.114 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:40:47.118 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:40:47.119 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:40:47.119 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:47.119 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:40:47.119 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:47.119 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:40:47.120 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:40:47.120 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:47.120 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:40:47.120 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:47.120 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:47.120 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:40:47.120 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:51.122 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:40:56.126 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:40:56.126 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:40:56.126 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:56.126 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:40:56.126 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:40:56.127 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:40:56.127 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:40:56.127 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:56.127 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:40:56.128 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:56.128 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:40:56.129 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:40:56.129 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:00.129 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:41:05.133 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:41:05.133 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:41:05.134 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:05.134 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:41:05.134 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:05.134 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:41:05.134 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:41:05.135 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:05.135 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:41:05.135 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:05.135 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:05.135 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:41:05.135 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:09.134 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:41:14.139 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:41:14.140 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:41:14.140 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:14.140 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:41:14.140 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:14.140 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:41:14.141 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:41:14.141 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:14.141 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:41:14.141 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:14.141 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:14.141 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:41:14.142 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:18.143 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:41:23.148 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:41:23.148 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:41:23.148 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:23.148 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:41:23.149 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:23.149 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:41:23.149 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:41:23.150 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:23.150 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:41:23.150 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:23.150 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:23.150 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:41:23.150 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:27.150 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:41:28.551 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:41:32.153 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:41:32.154 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:41:32.154 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:32.154 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:41:32.154 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:32.154 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:32.155 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51596712 disk_space_available_1k = 29493772 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:41:32.155 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:32.155 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:41:32.156 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:41:32.156 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:32.156 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:41:32.156 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:32.156 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:32.157 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:41:32.157 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:36.154 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:41:37.121 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51596952, partition_space_available_1k=29493532, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1316760, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18552, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48196, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14986568, free=9698432, buffers=681916, cached=7298200, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.45, five_min_avg=2.95, fifteen_min_avg=3.26, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:41:41.157 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:41:41.157 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:41:41.157 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:41.157 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:41:41.158 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:41.158 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:41.158 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51596952 partition_space_available_1k = 29493532 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1316760 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 18552 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 48196 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14986568 free = 9698432 buffers = 681916 cached = 7298200 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.45 five_min_avg = 2.95 fifteen_min_avg = 3.26 cpu_share = 2.75 node_type = database-node >> >> 10/21/2024 02:41:41.158 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:41.158 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:41:41.159 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:41:41.159 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:41.159 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:41:41.159 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:41.159 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:41.159 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:41:41.159 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:45.159 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:41:50.165 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:41:50.166 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:41:50.166 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:50.166 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:41:50.166 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:50.166 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:41:50.167 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:41:50.167 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:50.167 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:41:50.167 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:50.168 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:50.168 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:41:50.168 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:54.166 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:41:59.171 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:41:59.172 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:41:59.172 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:59.172 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:41:59.172 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:41:59.173 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:41:59.173 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:41:59.174 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:59.174 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:41:59.174 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:59.174 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:41:59.174 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:41:59.174 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:03.175 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:42:08.181 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:42:08.181 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:42:08.181 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:42:08.182 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:42:08.182 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:42:08.182 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:42:08.183 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:42:08.183 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:08.183 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:42:08.183 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:08.183 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:08.184 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:42:08.184 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:12.184 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:42:17.189 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:42:17.190 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:42:17.190 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:42:17.190 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:42:17.190 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:42:17.190 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:42:17.191 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:42:17.191 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:17.191 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:42:17.191 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:17.191 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:17.191 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:42:17.191 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:21.193 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:42:26.196 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:42:26.197 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:42:26.197 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:42:26.197 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:42:26.198 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:42:26.198 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:42:26.198 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:42:26.199 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:26.199 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:42:26.199 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:26.199 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:26.199 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:42:26.200 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:28.513 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:42:28.957 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:28.957 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51600028 disk_space_available_1k = 29490456 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:42:28.957 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:30.197 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:42:35.200 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:42:35.201 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:42:35.201 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:42:35.201 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:42:35.201 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:42:35.202 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:42:35.202 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:42:35.203 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:35.203 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:42:35.203 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:35.203 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:35.203 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:42:35.203 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:37.257 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51600264, partition_space_available_1k=29490220, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1316568, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18568, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48236, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14976652, free=9703812, buffers=681916, cached=7302736, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.92, five_min_avg=2.73, fifteen_min_avg=3.16, cpu_share=2.88, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:42:37.258 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:37.258 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51600264 partition_space_available_1k = 29490220 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1316568 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 18568 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 48236 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14976652 free = 9703812 buffers = 681916 cached = 7302736 node_type = database-node >> system_cpu_usage = << one_min_avg = 1.92 five_min_avg = 2.73 fifteen_min_avg = 3.16 cpu_share = 2.88 node_type = database-node >> >> 10/21/2024 02:42:37.259 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:39.205 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:42:44.215 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:42:44.215 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:42:44.215 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:42:44.215 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:42:44.215 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:42:44.216 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:42:44.216 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:42:44.216 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:44.216 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:42:44.218 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:44.218 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:44.218 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:42:44.218 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:48.218 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:42:53.222 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:42:53.223 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:42:53.223 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:42:53.223 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:42:53.223 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:42:53.223 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:42:53.224 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:42:53.224 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:53.224 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:42:53.224 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:53.224 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:53.224 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:42:53.225 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:42:57.225 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:43:02.230 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:43:02.230 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:43:02.231 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:02.231 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:43:02.231 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:02.231 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:43:02.232 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:43:02.232 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:02.232 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:43:02.232 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:02.232 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:02.232 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:43:02.233 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:06.232 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:43:11.235 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:43:11.235 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:43:11.235 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:11.235 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:43:11.235 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:11.236 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:43:11.236 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:43:11.236 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:11.237 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:43:11.237 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:11.237 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:11.237 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:43:11.237 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:15.237 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:43:20.242 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:43:20.243 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:43:20.243 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:20.243 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:43:20.243 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:20.243 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:43:20.244 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:43:20.244 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:20.244 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:43:20.244 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:20.244 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:20.244 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:43:20.244 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:24.246 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:43:28.503 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: dir for analytics is /var/lib/cassandra/data 10/21/2024 02:43:29.247 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:43:29.248 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:43:29.248 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:29.248 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:43:29.248 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:29.248 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:29.248 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] DatabaseUsage: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 database_usage = [ << disk_space_used_1k = 51603940 disk_space_available_1k = 29486544 analytics_db_size_1k = 916 >>, ] >> 10/21/2024 02:43:29.248 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:29.248 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:43:29.249 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:43:29.249 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:29.249 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:43:29.249 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:29.249 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:29.250 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:43:29.250 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:33.248 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:43:37.058 7fd1179c4f20 [contrail-database-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-439-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51599248, partition_space_available_1k=29491236, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'cassandra': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=1316596, cpu_share=0, mem_res=0, _EventManager__key='cassandra'), 'contrail-query-engine': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18572, cpu_share=0, mem_res=0, _EventManager__key='contrail-query-engine'), 'contrail-database-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=48276, cpu_share=0, mem_res=0, _EventManager__key='contrail-database-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665116, used=14957432, free=9722828, buffers=681916, cached=7302940, node_type='database-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.01, five_min_avg=2.65, fifteen_min_avg=3.11, cpu_share=2.75, node_type='database-node'), running_package_version=None, installed_package_version=None, _table='ObjectDatabaseInfo') 10/21/2024 02:43:38.251 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:43:38.251 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:43:38.251 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:38.252 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:43:38.252 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:38.252 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:38.252 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51599248 partition_space_available_1k = 29491236 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { cassandra : << mem_virt = 1316596 cpu_share = 0 mem_res = 0 >>contrail-query-engine : << mem_virt = 18572 cpu_share = 0 mem_res = 0 >>contrail-database-nodemgr : << mem_virt = 48276 cpu_share = 0 mem_res = 0 >> } system_mem_usage = << total = 32665116 used = 14957432 free = 9722828 buffers = 681916 cached = 7302940 node_type = database-node >> system_cpu_usage = << one_min_avg = 2.01 five_min_avg = 2.65 fifteen_min_avg = 3.11 cpu_share = 2.75 node_type = database-node >> >> 10/21/2024 02:43:38.252 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:38.252 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:43:38.253 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:43:38.253 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:38.253 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:43:38.253 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:38.253 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:38.254 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:43:38.254 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:42.253 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:43:47.258 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:43:47.258 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:43:47.258 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:47.259 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:43:47.259 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:47.259 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:43:47.260 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:43:47.260 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:47.260 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:43:47.261 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:47.261 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:47.261 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:43:47.261 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:51.261 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:43:56.264 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:43:56.265 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:43:56.265 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:56.265 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:43:56.265 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:43:56.266 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:43:56.267 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:43:56.267 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:56.267 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:43:56.267 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:56.267 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:56.268 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:43:56.268 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:43:56.277 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:43:57.273 7fd1112d6848 [contrail-database-nodemgr] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:NodeStatusUVE] map 10/21/2024 02:44:00.266 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 10/21/2024 02:44:05.269 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: Session Event: TCP Connect Fail 10/21/2024 02:44:05.269 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 10/21/2024 02:44:05.269 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:44:05.270 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 10/21/2024 02:44:05.270 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 10/21/2024 02:44:05.270 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 10/21/2024 02:44:05.271 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 10/21/2024 02:44:05.271 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:44:05.271 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-439-1 process_status = [ << module_id = contrail-database-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.30:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 10/21/2024 02:44:05.271 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:44:05.271 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:44:05.272 7fd1112d6848 [contrail-database-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-439-1:Database:contrail-database-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8103 start_time = 1729476087481404 collector_name = collector_ip = 10.0.0.30:8086 collector_list = [ 10.0.0.30:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 10/21/2024 02:44:05.272 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 10/21/2024 02:44:09.272 7fd1112d6848 [contrail-database-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle]