06/02/2025 02:05:58.127 7f6e75d6f308 [contrail-analytics-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 06/02/2025 02:05:58.128 7f6e75d6f308 [contrail-analytics-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 06/02/2025 02:05:58.128 7f6e75d6f308 [contrail-analytics-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 06/02/2025 02:05:58.140 7f6e7222b048 [contrail-analytics-nodemgr] [ERROR]: Cannot write http_port 8104 to /tmp/contrail-analytics-nodemgr.0.http_port 06/02/2025 02:05:58.141 7f6e7222b048 [contrail-analytics-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8104 06/02/2025 02:05:58.142 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvStart] in state[none] 06/02/2025 02:05:58.142 7f6e72230978 [contrail-analytics-nodemgr] [DEBUG]: Add uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:05:58.143 7f6e72230978 [contrail-analytics-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/02/2025 02:05:58.143 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 06/02/2025 02:05:58.143 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:05:58.143 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/02/2025 02:05:58.143 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:05:58.143 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:05:58.143 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 06/02/2025 02:05:58.143 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:05:58.183 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'contrail-collector', 'group': 'contrail-collector', 'pid': 17702134604529755006429219662402376059736787379561304959891443159406880401464, 'start': '1748829955000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-analytics-api', 'group': 'contrail-analytics-api', 'pid': 58858049705997164258307940887448948942332950755219282026806071143571702252597, 'start': '1748829955000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-analytics-nodemgr', 'group': 'contrail-analytics-nodemgr', 'pid': 3900803090225892158648066567564695298344253354413775266607402671234555305946, 'start': '1748829955000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 06/02/2025 02:05:58.457 7f6e75d6f308 [contrail-analytics-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-analytics-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-542-1', _node_type='Analytics', _instance_id='0', _timestamp=1748829958457379, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-analytics-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo'), _seqnum=0) 06/02/2025 02:05:58.458 7f6e75d6f308 [contrail-analytics-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-analytics-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-542-1', _node_type='Analytics', _instance_id='0', _timestamp=1748829958458099, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='contrail-collector', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1748829955000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-analytics-api', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1748829955000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-analytics-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1748829955000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-02 00:52:12.596288", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9311-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo'), _seqnum=0) 06/02/2025 02:05:58.461 7f6e7222b828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:05:58.462 7f6e7222b828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-02 00:52:12.596288", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9311-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 06/02/2025 02:05:58.462 7f6e7222b828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:05:58.462 7f6e7222b828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:05:58.462 7f6e7222b828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/02/2025 02:05:58.463 7f6e7222b828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:05:58.463 7f6e7222b828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:05:58.463 7f6e7222b828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 deleted = False process_info = [ << process_name = contrail-collector process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1748829955000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-analytics-api process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1748829955000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-analytics-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1748829955000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-02 00:52:12.596288", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9311-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 06/02/2025 02:05:58.463 7f6e7222b828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:02.144 7f6e72dc2ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:06:03.491 7f6e75d6f308 [contrail-analytics-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-analytics-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-542-1', _node_type='Analytics', _instance_id='0', _timestamp=1748829963491073, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='contrail-collector', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1748829955000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-analytics-api', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1748829955000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-analytics-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1748829955000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-02 00:52:12.596288", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9311-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo'), _seqnum=0) 06/02/2025 02:06:03.507 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=50851404, partition_space_available_1k=30239080, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19892, cpu_share=0.0, mem_res=4084, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=6204, cpu_share=0.0, mem_res=800, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=81536, cpu_share=0.0, mem_res=39608, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=9491492, free=1090188, buffers=959624, cached=21123816, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.66, five_min_avg=3.21, fifteen_min_avg=2.41, cpu_share=0, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:06:07.151 7f6e72dc2ac8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:06:07.151 7f6e72dc2ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:06:07.151 7f6e72dc2ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:07.152 7f6e72dc2ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:06:07.152 7f6e72dc2ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:07.152 7f6e72dc2ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:07.152 7f6e72dc2ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 deleted = False process_info = [ << process_name = contrail-collector process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1748829955000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-analytics-api process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1748829955000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-analytics-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1748829955000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-02 00:52:12.596288", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9311-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 06/02/2025 02:06:07.153 7f6e72dc2ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:07.153 7f6e72dc2ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:07.153 7f6e72dc2ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 50851404 partition_space_available_1k = 30239080 percentage_partition_space_used = 63 >> } all_core_file_list = [ ] process_mem_cpu_usage = { contrail-collector : << mem_virt = 19892 cpu_share = 0.0 mem_res = 4084 >>contrail-analytics-api : << mem_virt = 6204 cpu_share = 0.0 mem_res = 800 >>contrail-analytics-nodemgr : << mem_virt = 81536 cpu_share = 0.0 mem_res = 39608 >> } system_mem_usage = << total = 32665120 used = 9491492 free = 1090188 buffers = 959624 cached = 21123816 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 3.66 five_min_avg = 3.21 fifteen_min_avg = 2.41 cpu_share = 0 node_type = analytics-node >> >> 06/02/2025 02:06:07.153 7f6e72dc2ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:07.153 7f6e72dc2ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:06:07.154 7f6e72dc2ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:06:07.154 7f6e72dc2ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:07.154 7f6e72dc2ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:06:07.154 7f6e72dc2ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:07.154 7f6e72dc2ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:07.155 7f6e72dc2ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:06:07.155 7f6e72dc2ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:11.153 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:06:11.354 7f6e72dc2c18 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:06:16.157 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:06:16.157 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:06:16.157 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:16.158 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:06:16.158 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:16.158 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:06:16.158 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:06:16.159 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:16.159 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:06:16.159 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:16.159 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:16.159 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:06:16.159 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:20.161 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:06:20.886 7f6e72dc5d68 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:06:25.168 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:06:25.168 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:06:25.169 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:25.169 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:06:25.169 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:25.169 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:06:25.170 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:06:25.170 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:25.170 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:06:25.170 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:25.170 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:25.170 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:06:25.170 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:29.169 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:06:31.925 7f6e7222ba20 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:06:34.174 7f6e7366b0f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:06:34.175 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:06:34.175 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:34.175 7f6e7366b0f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:06:34.175 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:34.175 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:06:34.175 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:06:34.176 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:34.176 7f6e7366b0f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:06:34.176 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:34.176 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:34.176 7f6e7366b0f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:06:34.176 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:38.179 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:06:43.189 7f6e7366b0f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:06:43.197 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:06:43.197 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:43.197 7f6e7366b0f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:06:43.197 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:43.197 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:06:43.198 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:06:43.198 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:43.198 7f6e7366b0f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:06:43.198 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:43.198 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:43.198 7f6e7366b0f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:06:43.198 7f6e7366b0f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:47.190 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:06:52.197 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:06:52.197 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:06:52.198 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:52.198 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:06:52.198 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:52.198 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:06:52.199 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:06:52.199 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:52.199 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:06:52.199 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:52.200 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:52.201 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:06:52.201 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:52.479 7f6e7369a438 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:06:56.201 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:07:01.218 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:07:01.219 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:07:01.219 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:01.219 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:07:01.219 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:01.219 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:07:01.220 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:07:01.220 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:01.220 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:07:01.220 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:01.220 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:01.220 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:07:01.220 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:03.500 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51205760, partition_space_available_1k=29884724, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=28432, cpu_share=0.06, mem_res=12556, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=5544, cpu_share=0.0, mem_res=800, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=58720, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=13945608, free=12850444, buffers=951148, cached=4917920, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=9.36, five_min_avg=4.75, fifteen_min_avg=2.98, cpu_share=8.0, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:07:03.500 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:03.501 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51205760 partition_space_available_1k = 29884724 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 28432 cpu_share = 0.06 mem_res = 12556 >>contrail-analytics-api : << mem_virt = 5544 cpu_share = 0.0 mem_res = 800 >>contrail-analytics-nodemgr : << mem_virt = 58720 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 13945608 free = 12850444 buffers = 951148 cached = 4917920 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 9.36 five_min_avg = 4.75 fifteen_min_avg = 2.98 cpu_share = 8.0 node_type = analytics-node >> >> 06/02/2025 02:07:03.501 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:04.802 7f6e7369a438 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:07:05.218 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:07:10.225 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:07:10.229 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:07:10.229 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:10.229 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:07:10.229 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:10.229 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:07:10.230 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:07:10.230 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:10.230 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:07:10.230 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:10.230 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:10.230 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:07:10.230 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:14.229 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:07:18.290 7f6e75b14780 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:07:19.234 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:07:19.234 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:07:19.234 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:19.234 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:07:19.234 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:19.234 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:07:19.244 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:07:19.244 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:19.281 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:07:19.281 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:19.281 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:19.281 7f6e7369a438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:07:19.281 7f6e7369a438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:23.239 7f6e75d04048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:07:28.246 7f6e75d04048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:07:28.247 7f6e75d04048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:07:28.247 7f6e75d04048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:28.247 7f6e75d04048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:07:28.247 7f6e75d04048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:28.247 7f6e75d04048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:07:28.247 7f6e75d04048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:07:28.247 7f6e75d04048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:28.248 7f6e75d04048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:07:28.248 7f6e75d04048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:28.248 7f6e75d04048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:28.248 7f6e75d04048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:07:28.248 7f6e75d04048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:32.249 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:07:37.257 7f6e7369a4e0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:07:37.257 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:07:37.257 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:37.258 7f6e7369a4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:07:37.258 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:37.258 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:07:37.258 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:07:37.259 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:37.259 7f6e7369a4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:07:37.259 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:37.260 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:37.260 7f6e7369a4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:07:37.260 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:40.878 7f6e7369a4e0 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:07:41.257 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:07:46.284 7f6e7369a4e0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:07:46.284 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:07:46.284 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:46.284 7f6e7369a4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:07:46.284 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:46.285 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:07:46.285 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:07:46.285 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:46.285 7f6e7369a4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:07:46.286 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:46.286 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:46.286 7f6e7369a4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:07:46.286 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:50.286 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:07:55.293 7f6e72245e10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:07:55.293 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:07:55.294 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:55.294 7f6e72245e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:07:55.294 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:55.295 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:07:55.295 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:07:55.295 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:55.295 7f6e72245e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:07:55.295 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:55.295 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:55.295 7f6e72245e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:07:55.296 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:59.295 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:08:03.504 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51251580, partition_space_available_1k=29838904, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29388, cpu_share=0.0, mem_res=13512, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=103812, cpu_share=0.06, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=58800, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=14803748, free=10626188, buffers=987144, cached=6248040, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=19.69, five_min_avg=8.97, fifteen_min_avg=4.56, cpu_share=10.0, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:08:04.299 7f6e72245e10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:08:04.299 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:08:04.299 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:04.299 7f6e72245e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:08:04.299 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:04.299 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:04.300 7f6e72245e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51251580 partition_space_available_1k = 29838904 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29388 cpu_share = 0.0 mem_res = 13512 >>contrail-analytics-api : << mem_virt = 103812 cpu_share = 0.06 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 58800 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 14803748 free = 10626188 buffers = 987144 cached = 6248040 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 19.69 five_min_avg = 8.97 fifteen_min_avg = 4.56 cpu_share = 10.0 node_type = analytics-node >> >> 06/02/2025 02:08:04.300 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:04.300 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:08:04.300 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:08:04.300 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:04.300 7f6e72245e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:08:04.300 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:04.301 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:04.301 7f6e72245e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:08:04.301 7f6e72245e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:08.300 7f6e72e25438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:08:13.306 7f6e72e25438 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:08:13.306 7f6e72e25438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:08:13.306 7f6e72e25438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:13.307 7f6e72e25438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:08:13.307 7f6e72e25438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:13.307 7f6e72e25438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:08:13.307 7f6e72e25438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:08:13.308 7f6e72e25438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:13.308 7f6e72e25438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:08:13.308 7f6e72e25438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:13.308 7f6e72e25438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:13.308 7f6e72e25438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:08:13.308 7f6e72e25438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:15.387 7f6e75d04048 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:08:17.308 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:08:22.314 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:08:22.314 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:08:22.314 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:22.314 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:08:22.314 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:22.314 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:08:22.315 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:08:22.315 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:22.315 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:08:22.315 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:22.315 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:22.315 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:08:22.316 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:26.318 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:08:30.638 7f6e77207eb8 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:08:31.321 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:08:31.321 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:08:31.321 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:31.321 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:08:31.321 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:31.321 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:08:31.322 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:08:31.323 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:31.323 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:08:31.323 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:31.323 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:31.324 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:08:31.324 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:35.324 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:08:40.330 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:08:40.331 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:08:40.331 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:40.331 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:08:40.332 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:40.332 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:08:40.332 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:08:40.332 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:40.332 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:08:40.332 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:40.333 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:40.333 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:08:40.333 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:44.336 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:08:49.369 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:08:49.370 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:08:49.370 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:49.370 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:08:49.370 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:49.370 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:08:49.370 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:08:49.370 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:49.371 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:08:49.371 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:49.371 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:49.371 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:08:49.371 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:52.371 7f6e756aa978 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:08:53.370 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:08:58.375 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:08:58.375 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:08:58.376 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:58.376 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:08:58.376 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:58.376 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:08:58.376 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:08:58.376 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:58.376 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:08:58.377 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:58.377 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:58.377 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:08:58.377 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:02.377 7f6e77201f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:09:02.803 7f6e766dbc18 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:09:03.494 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51279560, partition_space_available_1k=29810924, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29388, cpu_share=0.0, mem_res=13512, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=93832, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=58832, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15159504, free=11305868, buffers=990816, cached=5208932, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=12.46, five_min_avg=8.85, fifteen_min_avg=4.8, cpu_share=5.38, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:09:07.382 7f6e77201f60 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:09:07.383 7f6e77201f60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:09:07.383 7f6e77201f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:07.384 7f6e77201f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:09:07.384 7f6e77201f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:07.384 7f6e77201f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:07.384 7f6e77201f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51279560 partition_space_available_1k = 29810924 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29388 cpu_share = 0.0 mem_res = 13512 >>contrail-analytics-api : << mem_virt = 93832 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 58832 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15159504 free = 11305868 buffers = 990816 cached = 5208932 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 12.46 five_min_avg = 8.85 fifteen_min_avg = 4.8 cpu_share = 5.38 node_type = analytics-node >> >> 06/02/2025 02:09:07.384 7f6e77201f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:07.384 7f6e77201f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:09:07.385 7f6e77201f60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:09:07.385 7f6e77201f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:07.385 7f6e77201f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:09:07.385 7f6e77201f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:07.385 7f6e77201f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:07.385 7f6e77201f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:09:07.385 7f6e77201f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:11.387 7f6e7368b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:09:16.389 7f6e7368b390 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:09:16.389 7f6e7368b390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:09:16.389 7f6e7368b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:16.390 7f6e7368b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:09:16.390 7f6e7368b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:16.390 7f6e7368b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:09:16.390 7f6e7368b390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:09:16.390 7f6e7368b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:16.391 7f6e7368b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:09:16.391 7f6e7368b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:16.391 7f6e7368b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:16.391 7f6e7368b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:09:16.391 7f6e7368b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:20.392 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:09:25.398 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:09:25.398 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:09:25.398 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:25.398 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:09:25.398 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:25.398 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:09:25.399 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:09:25.399 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:25.399 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:09:25.399 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:25.399 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:25.400 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:09:25.400 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:29.401 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:09:34.405 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:09:34.405 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:09:34.406 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:34.406 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:09:34.406 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:34.406 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:09:34.406 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:09:34.406 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:34.407 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:09:34.407 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:34.407 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:34.407 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:09:34.407 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:38.408 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:09:43.419 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:09:43.419 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:09:43.419 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:43.419 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:09:43.419 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:43.420 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:09:43.421 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:09:43.421 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:43.421 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:09:43.421 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:43.421 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:43.421 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:09:43.421 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:47.418 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:09:52.425 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:09:52.425 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:09:52.425 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:52.425 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:09:52.425 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:52.425 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:09:52.426 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:09:52.426 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:52.426 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:09:52.426 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:52.426 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:52.426 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:09:52.426 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:56.429 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:10:01.433 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:10:01.433 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:10:01.433 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:01.434 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:10:01.434 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:01.434 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:10:01.434 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:10:01.435 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:01.435 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:10:01.435 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:01.435 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:01.435 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:10:01.435 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:03.499 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51287160, partition_space_available_1k=29803324, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29396, cpu_share=0.0, mem_res=13512, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=93836, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=58828, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15116004, free=11168700, buffers=990888, cached=5389528, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.45, five_min_avg=7.78, fifteen_min_avg=4.68, cpu_share=3.88, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:10:03.499 7f6e77208588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:03.500 7f6e77208588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51287160 partition_space_available_1k = 29803324 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29396 cpu_share = 0.0 mem_res = 13512 >>contrail-analytics-api : << mem_virt = 93836 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 58828 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15116004 free = 11168700 buffers = 990888 cached = 5389528 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 6.45 five_min_avg = 7.78 fifteen_min_avg = 4.68 cpu_share = 3.88 node_type = analytics-node >> >> 06/02/2025 02:10:03.500 7f6e77208588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:05.435 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:10:10.441 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:10:10.441 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:10:10.441 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:10.441 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:10:10.441 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:10.441 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:10:10.442 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:10:10.442 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:10.442 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:10:10.442 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:10.442 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:10.442 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:10:10.442 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:14.445 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:10:19.449 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:10:19.449 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:10:19.450 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:19.450 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:10:19.450 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:19.450 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:10:19.451 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:10:19.451 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:19.451 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:10:19.451 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:19.451 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:19.451 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:10:19.451 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:23.452 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:10:23.802 7f6e7368b0f0 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:10:28.457 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:10:28.457 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:10:28.457 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:28.457 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:10:28.457 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:28.457 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:10:28.458 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:10:28.458 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:28.458 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:10:28.458 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:28.458 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:28.458 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:10:28.458 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:32.457 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:10:34.461 7f6e736a7630 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:10:37.462 7f6e72229e10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:10:37.462 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:10:37.462 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:37.462 7f6e72229e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:10:37.462 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:37.462 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:10:37.463 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:10:37.463 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:37.463 7f6e72229e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:10:37.463 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:37.463 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:37.463 7f6e72229e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:10:37.463 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:41.466 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:10:46.473 7f6e72229e10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:10:46.473 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:10:46.474 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:46.474 7f6e72229e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:10:46.474 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:46.474 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:10:46.475 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:10:46.475 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:46.476 7f6e72229e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:10:46.476 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:46.476 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:46.476 7f6e72229e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:10:46.476 7f6e72229e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:50.477 7f6e7b9b9240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:10:55.482 7f6e7b9b9240 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:10:55.482 7f6e7b9b9240 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:10:55.482 7f6e7b9b9240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:55.482 7f6e7b9b9240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:10:55.482 7f6e7b9b9240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:55.482 7f6e7b9b9240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:10:55.483 7f6e7b9b9240 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:10:55.483 7f6e7b9b9240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:55.483 7f6e7b9b9240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:10:55.483 7f6e7b9b9240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:55.483 7f6e7b9b9240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:55.483 7f6e7b9b9240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:10:55.483 7f6e7b9b9240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:59.484 7f6e72dc3e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:11:03.518 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51290568, partition_space_available_1k=29799916, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29468, cpu_share=0.0, mem_res=13588, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=93864, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59224, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15169112, free=11074504, buffers=991568, cached=5429936, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.47, five_min_avg=6.9, fifteen_min_avg=4.57, cpu_share=3.12, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:11:04.486 7f6e72dc3e10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:11:04.487 7f6e72dc3e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:11:04.487 7f6e72dc3e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:04.487 7f6e72dc3e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:11:04.487 7f6e72dc3e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:04.487 7f6e72dc3e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:04.487 7f6e72dc3e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51290568 partition_space_available_1k = 29799916 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29468 cpu_share = 0.0 mem_res = 13588 >>contrail-analytics-api : << mem_virt = 93864 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59224 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15169112 free = 11074504 buffers = 991568 cached = 5429936 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 4.47 five_min_avg = 6.9 fifteen_min_avg = 4.57 cpu_share = 3.12 node_type = analytics-node >> >> 06/02/2025 02:11:04.487 7f6e72dc3e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:04.488 7f6e72dc3e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:11:04.488 7f6e72dc3e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:11:04.488 7f6e72dc3e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:04.488 7f6e72dc3e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:11:04.488 7f6e72dc3e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:04.488 7f6e72dc3e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:04.489 7f6e72dc3e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:11:04.489 7f6e72dc3e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:05.735 7f6e7af05f60 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:11:08.489 7f6e7af05f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:11:13.493 7f6e7af05f60 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:11:13.493 7f6e7af05f60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:11:13.493 7f6e7af05f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:13.493 7f6e7af05f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:11:13.493 7f6e7af05f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:13.493 7f6e7af05f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:11:13.494 7f6e7af05f60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:11:13.494 7f6e7af05f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:13.494 7f6e7af05f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:11:13.494 7f6e7af05f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:13.494 7f6e7af05f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:13.494 7f6e7af05f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:11:13.494 7f6e7af05f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:17.493 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:11:22.499 7f6e73a77780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:11:22.499 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:11:22.500 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:22.500 7f6e73a77780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:11:22.500 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:22.500 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:11:22.500 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:11:22.501 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:22.501 7f6e73a77780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:11:22.501 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:22.501 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:22.501 7f6e73a77780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:11:22.501 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:26.502 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:11:31.509 7f6e73a77780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:11:31.509 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:11:31.509 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:31.509 7f6e73a77780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:11:31.509 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:31.509 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:11:31.509 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:11:31.510 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:31.510 7f6e73a77780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:11:31.510 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:31.510 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:31.510 7f6e73a77780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:11:31.510 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:35.512 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:11:40.517 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:11:40.518 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:11:40.518 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:40.518 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:11:40.518 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:40.518 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:11:40.518 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:11:40.518 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:40.518 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:11:40.519 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:40.519 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:40.519 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:11:40.519 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:44.520 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:11:49.524 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:11:49.524 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:11:49.524 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:49.525 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:11:49.525 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:49.525 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:11:49.525 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:11:49.526 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:49.526 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:11:49.526 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:49.526 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:49.526 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:11:49.526 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:53.524 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:11:56.091 7f6e72dbaeb8 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:11:58.526 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:11:58.527 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:11:58.527 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:58.527 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:11:58.527 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:58.527 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:11:58.528 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:11:58.529 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:58.529 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:11:58.529 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:58.529 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:58.529 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:11:58.529 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:02.528 7f6e73673cc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:12:03.506 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51302484, partition_space_available_1k=29788000, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29468, cpu_share=0.0, mem_res=13588, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=93932, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59244, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15108140, free=11105268, buffers=991980, cached=5459732, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.39, five_min_avg=6.19, fifteen_min_avg=4.47, cpu_share=2.88, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:12:07.534 7f6e73673cc0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:12:07.534 7f6e73673cc0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:12:07.534 7f6e73673cc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:07.534 7f6e73673cc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:12:07.535 7f6e73673cc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:07.535 7f6e73673cc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:07.535 7f6e73673cc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51302484 partition_space_available_1k = 29788000 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29468 cpu_share = 0.0 mem_res = 13588 >>contrail-analytics-api : << mem_virt = 93932 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59244 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15108140 free = 11105268 buffers = 991980 cached = 5459732 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 3.39 five_min_avg = 6.19 fifteen_min_avg = 4.47 cpu_share = 2.88 node_type = analytics-node >> >> 06/02/2025 02:12:07.535 7f6e73673cc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:07.535 7f6e73673cc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:12:07.535 7f6e73673cc0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:12:07.536 7f6e73673cc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:07.536 7f6e73673cc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:12:07.536 7f6e73673cc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:07.536 7f6e73673cc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:07.536 7f6e73673cc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:12:07.536 7f6e73673cc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:11.541 7f6e771f4eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:12:16.550 7f6e771f4eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:12:16.550 7f6e771f4eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:12:16.550 7f6e771f4eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:16.550 7f6e771f4eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:12:16.550 7f6e771f4eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:16.550 7f6e771f4eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:12:16.551 7f6e771f4eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:12:16.551 7f6e771f4eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:16.551 7f6e771f4eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:12:16.551 7f6e771f4eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:16.551 7f6e771f4eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:16.551 7f6e771f4eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:12:16.551 7f6e771f4eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:20.550 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:12:25.557 7f6e72e286d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:12:25.557 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:12:25.557 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:25.558 7f6e72e286d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:12:25.558 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:25.558 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:12:25.558 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:12:25.558 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:25.558 7f6e72e286d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:12:25.559 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:25.559 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:25.559 7f6e72e286d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:12:25.559 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:29.561 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:12:34.564 7f6e72e286d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:12:34.564 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:12:34.564 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:34.564 7f6e72e286d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:12:34.564 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:34.564 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:12:34.565 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:12:34.565 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:34.565 7f6e72e286d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:12:34.565 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:34.565 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:34.565 7f6e72e286d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:12:34.565 7f6e72e286d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:38.567 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:12:43.573 7f6e771d62e8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:12:43.573 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:12:43.573 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:43.573 7f6e771d62e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:12:43.573 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:43.573 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:12:43.573 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:12:43.574 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:43.574 7f6e771d62e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:12:43.574 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:43.574 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:43.574 7f6e771d62e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:12:43.574 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:47.576 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:12:52.582 7f6e771d62e8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:12:52.582 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:12:52.582 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:52.582 7f6e771d62e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:12:52.582 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:52.582 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:12:52.583 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:12:52.583 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:52.583 7f6e771d62e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:12:52.583 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:52.583 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:52.583 7f6e771d62e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:12:52.583 7f6e771d62e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:56.587 7f6e72dc2eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:13:01.589 7f6e72dc2eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:13:01.589 7f6e72dc2eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:13:01.589 7f6e72dc2eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:01.590 7f6e72dc2eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:13:01.590 7f6e72dc2eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:01.590 7f6e72dc2eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:13:01.590 7f6e72dc2eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:13:01.591 7f6e72dc2eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:01.591 7f6e72dc2eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:13:01.591 7f6e72dc2eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:01.591 7f6e72dc2eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:01.591 7f6e72dc2eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:13:01.591 7f6e72dc2eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:03.501 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51314856, partition_space_available_1k=29775628, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29468, cpu_share=0.0, mem_res=13588, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=93928, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59248, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15136000, free=9738536, buffers=991988, cached=6798596, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.48, five_min_avg=6.59, fifteen_min_avg=4.73, cpu_share=5.25, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:13:03.501 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:03.501 7f6e722426d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51314856 partition_space_available_1k = 29775628 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29468 cpu_share = 0.0 mem_res = 13588 >>contrail-analytics-api : << mem_virt = 93928 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59248 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15136000 free = 9738536 buffers = 991988 cached = 6798596 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 5.48 five_min_avg = 6.59 fifteen_min_avg = 4.73 cpu_share = 5.25 node_type = analytics-node >> >> 06/02/2025 02:13:03.501 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:05.592 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:13:10.598 7f6e771fb0f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:13:10.599 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:13:10.599 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:10.599 7f6e771fb0f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:13:10.599 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:10.599 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:13:10.599 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:13:10.599 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:10.600 7f6e771fb0f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:13:10.600 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:10.600 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:10.600 7f6e771fb0f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:13:10.600 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:14.602 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:13:19.604 7f6e771fb0f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:13:19.605 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:13:19.605 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:19.605 7f6e771fb0f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:13:19.605 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:19.605 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:13:19.605 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:13:19.605 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:19.606 7f6e771fb0f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:13:19.606 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:19.606 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:19.606 7f6e771fb0f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:13:19.606 7f6e771fb0f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:23.609 7f6e73698438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:13:28.614 7f6e73698438 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:13:28.615 7f6e73698438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:13:28.615 7f6e73698438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:28.615 7f6e73698438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:13:28.615 7f6e73698438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:28.615 7f6e73698438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:13:28.616 7f6e73698438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:13:28.616 7f6e73698438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:28.616 7f6e73698438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:13:28.616 7f6e73698438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:28.616 7f6e73698438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:28.616 7f6e73698438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:13:28.616 7f6e73698438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:32.616 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:13:37.621 7f6e75682390 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:13:37.621 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:13:37.621 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:37.622 7f6e75682390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:13:37.622 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:37.622 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:13:37.622 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:13:37.622 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:37.622 7f6e75682390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:13:37.623 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:37.623 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:37.623 7f6e75682390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:13:37.623 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:41.625 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:13:46.630 7f6e75682390 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:13:46.631 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:13:46.631 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:46.631 7f6e75682390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:13:46.631 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:46.631 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:13:46.631 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:13:46.631 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:46.632 7f6e75682390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:13:46.632 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:46.632 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:46.632 7f6e75682390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:13:46.632 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:50.632 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:13:55.637 7f6e73a77780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:13:55.637 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:13:55.637 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:55.637 7f6e73a77780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:13:55.637 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:55.637 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:13:55.638 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:13:55.638 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:55.638 7f6e73a77780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:13:55.638 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:55.638 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:55.638 7f6e73a77780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:13:55.638 7f6e73a77780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:59.638 7f6e771fdd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:14:03.505 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51321460, partition_space_available_1k=29769024, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29468, cpu_share=0.0, mem_res=13588, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=93960, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59288, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15161716, free=9705132, buffers=992000, cached=6806272, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.01, five_min_avg=5.96, fifteen_min_avg=4.63, cpu_share=2.62, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:14:04.640 7f6e771fdd68 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:14:04.641 7f6e771fdd68 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:14:04.641 7f6e771fdd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:04.641 7f6e771fdd68 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:14:04.641 7f6e771fdd68 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:04.641 7f6e771fdd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:04.641 7f6e771fdd68 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51321460 partition_space_available_1k = 29769024 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29468 cpu_share = 0.0 mem_res = 13588 >>contrail-analytics-api : << mem_virt = 93960 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59288 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15161716 free = 9705132 buffers = 992000 cached = 6806272 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 4.01 five_min_avg = 5.96 fifteen_min_avg = 4.63 cpu_share = 2.62 node_type = analytics-node >> >> 06/02/2025 02:14:04.641 7f6e771fdd68 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:04.641 7f6e771fdd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:14:04.642 7f6e771fdd68 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:14:04.643 7f6e771fdd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:04.644 7f6e771fdd68 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:14:04.644 7f6e771fdd68 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:04.644 7f6e771fdd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:04.644 7f6e771fdd68 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:14:04.644 7f6e771fdd68 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:08.645 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:14:13.649 7f6e722426d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:14:13.649 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:14:13.649 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:13.649 7f6e722426d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:14:13.649 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:13.649 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:14:13.650 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:14:13.650 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:13.650 7f6e722426d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:14:13.650 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:13.650 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:13.650 7f6e722426d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:14:13.650 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:17.652 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:14:22.657 7f6e722426d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:14:22.657 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:14:22.657 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:22.657 7f6e722426d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:14:22.657 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:22.657 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:14:22.658 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:14:22.658 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:22.658 7f6e722426d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:14:22.658 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:22.658 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:22.658 7f6e722426d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:14:22.658 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:26.661 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:14:31.666 7f6e722426d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:14:31.667 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:14:31.667 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:31.667 7f6e722426d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:14:31.667 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:31.667 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:14:31.668 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:14:31.668 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:31.668 7f6e722426d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:14:31.668 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:31.668 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:31.668 7f6e722426d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:14:31.668 7f6e722426d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:35.668 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:14:40.670 7f6e771eee10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:14:40.670 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:14:40.670 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:40.670 7f6e771eee10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:14:40.670 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:40.670 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:14:40.671 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:14:40.671 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:40.671 7f6e771eee10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:14:40.671 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:40.671 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:40.671 7f6e771eee10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:14:40.671 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:44.672 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:14:49.674 7f6e771eee10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:14:49.675 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:14:49.675 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:49.675 7f6e771eee10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:14:49.675 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:49.675 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:14:49.676 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:14:49.676 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:49.676 7f6e771eee10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:14:49.676 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:49.676 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:49.676 7f6e771eee10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:14:49.676 7f6e771eee10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:53.679 7f6e7222ba20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:14:58.681 7f6e7222ba20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:14:58.681 7f6e7222ba20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:14:58.681 7f6e7222ba20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:58.682 7f6e7222ba20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:14:58.682 7f6e7222ba20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:58.682 7f6e7222ba20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:14:58.682 7f6e7222ba20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:14:58.682 7f6e7222ba20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:58.683 7f6e7222ba20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:14:58.683 7f6e7222ba20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:58.683 7f6e7222ba20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:58.683 7f6e7222ba20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:14:58.683 7f6e7222ba20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:02.683 7f6e73698048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:15:03.512 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51327276, partition_space_available_1k=29763208, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29472, cpu_share=0.0, mem_res=13588, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94020, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59312, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15219988, free=9630504, buffers=992132, cached=6822496, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.63, five_min_avg=5.18, fifteen_min_avg=4.45, cpu_share=3.0, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:15:07.745 7f6e73698048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:15:07.745 7f6e73698048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:15:07.745 7f6e73698048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:07.745 7f6e73698048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:15:07.745 7f6e73698048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:07.745 7f6e73698048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:07.745 7f6e73698048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51327276 partition_space_available_1k = 29763208 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29472 cpu_share = 0.0 mem_res = 13588 >>contrail-analytics-api : << mem_virt = 94020 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59312 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15219988 free = 9630504 buffers = 992132 cached = 6822496 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 2.63 five_min_avg = 5.18 fifteen_min_avg = 4.45 cpu_share = 3.0 node_type = analytics-node >> >> 06/02/2025 02:15:07.745 7f6e73698048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:07.745 7f6e73698048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:15:07.746 7f6e73698048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:15:07.746 7f6e73698048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:07.746 7f6e73698048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:15:07.746 7f6e73698048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:07.746 7f6e73698048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:07.746 7f6e73698048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:15:07.746 7f6e73698048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:10.130 7f6e72dba588 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:15:11.747 7f6e72dba588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:15:16.753 7f6e72dba588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:15:16.753 7f6e72dba588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:15:16.754 7f6e72dba588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:16.754 7f6e72dba588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:15:16.754 7f6e72dba588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:16.754 7f6e72dba588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:15:16.755 7f6e72dba588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:15:16.755 7f6e72dba588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:16.755 7f6e72dba588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:15:16.755 7f6e72dba588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:16.755 7f6e72dba588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:16.755 7f6e72dba588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:15:16.755 7f6e72dba588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:20.444 7f6e736a7198 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:15:20.754 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:15:25.757 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:15:25.757 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:15:25.757 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:25.757 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:15:25.757 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:25.757 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:15:25.758 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:15:25.758 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:25.758 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:15:25.758 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:25.758 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:25.758 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:15:25.758 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:29.761 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:15:31.204 7f6e771dd8d0 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:15:34.766 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:15:34.767 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:15:34.767 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:34.767 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:15:34.767 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:34.767 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:15:34.768 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:15:34.768 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:34.768 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:15:34.768 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:34.768 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:34.768 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:15:34.768 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:38.770 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:15:43.776 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:15:43.776 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:15:43.777 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:43.777 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:15:43.777 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:43.777 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:15:43.777 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:15:43.777 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:43.777 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:15:43.777 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:43.777 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:43.778 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:15:43.778 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:47.781 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:15:52.785 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:15:52.785 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:15:52.786 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:52.786 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:15:52.786 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:52.786 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:15:52.787 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:15:52.787 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:52.787 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:15:52.787 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:52.787 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:52.788 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:15:52.788 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:56.788 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:16:01.791 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:16:01.791 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:16:01.791 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:01.792 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:16:01.792 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:01.792 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:16:01.792 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:16:01.793 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:01.793 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:16:01.793 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:01.793 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:01.793 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:16:01.793 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:03.522 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51336704, partition_space_available_1k=29753780, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29480, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94024, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59304, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15167656, free=9665240, buffers=993384, cached=6838840, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.5, five_min_avg=4.73, fifteen_min_avg=4.34, cpu_share=3.12, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:16:03.522 7f6e771ddd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:03.522 7f6e771ddd68 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51336704 partition_space_available_1k = 29753780 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29480 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94024 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59304 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15167656 free = 9665240 buffers = 993384 cached = 6838840 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 2.5 five_min_avg = 4.73 fifteen_min_avg = 4.34 cpu_share = 3.12 node_type = analytics-node >> >> 06/02/2025 02:16:03.523 7f6e771ddd68 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:05.791 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:16:10.798 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:16:10.798 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:16:10.798 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:10.798 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:16:10.798 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:10.799 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:16:10.799 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:16:10.800 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:10.800 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:16:10.800 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:10.800 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:10.800 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:16:10.800 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:14.800 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:16:19.805 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:16:19.805 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:16:19.805 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:19.805 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:16:19.805 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:19.806 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:16:19.806 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:16:19.806 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:19.806 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:16:19.806 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:19.806 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:19.807 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:16:19.807 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:23.819 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:16:28.831 7f6e77208198 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:16:28.831 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:16:28.832 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:28.832 7f6e77208198 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:16:28.832 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:28.832 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:16:28.832 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:16:28.833 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:28.833 7f6e77208198 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:16:28.833 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:28.833 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:28.833 7f6e77208198 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:16:28.833 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:32.834 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:16:37.839 7f6e77208198 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:16:37.839 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:16:37.839 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:37.839 7f6e77208198 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:16:37.839 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:37.839 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:16:37.840 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:16:37.840 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:37.840 7f6e77208198 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:16:37.840 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:37.840 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:37.840 7f6e77208198 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:16:37.840 7f6e77208198 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:41.843 7f6e7366b828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:16:46.850 7f6e7366b828 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:16:46.850 7f6e7366b828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:16:46.850 7f6e7366b828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:46.850 7f6e7366b828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:16:46.850 7f6e7366b828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:46.850 7f6e7366b828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:16:46.851 7f6e7366b828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:16:46.851 7f6e7366b828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:46.851 7f6e7366b828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:16:46.851 7f6e7366b828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:46.852 7f6e7366b828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:46.852 7f6e7366b828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:16:46.852 7f6e7366b828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:50.849 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:16:54.839 7f6e72dc5b70 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:16:55.852 7f6e75682390 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:16:55.852 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:16:55.853 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:55.853 7f6e75682390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:16:55.853 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:55.853 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:16:55.853 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:16:55.854 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:55.854 7f6e75682390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:16:55.854 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:55.854 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:55.854 7f6e75682390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:16:55.854 7f6e75682390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:59.853 7f6e73694048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:17:03.517 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51348908, partition_space_available_1k=29741576, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29476, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94052, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59356, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15184248, free=9607548, buffers=994116, cached=6879208, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.21, five_min_avg=5.21, fifteen_min_avg=4.53, cpu_share=5.25, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:17:04.857 7f6e73694048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:17:04.857 7f6e73694048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:17:04.857 7f6e73694048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:04.857 7f6e73694048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:17:04.858 7f6e73694048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:04.858 7f6e73694048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:04.858 7f6e73694048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51348908 partition_space_available_1k = 29741576 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29476 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94052 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59356 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15184248 free = 9607548 buffers = 994116 cached = 6879208 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 5.21 five_min_avg = 5.21 fifteen_min_avg = 4.53 cpu_share = 5.25 node_type = analytics-node >> >> 06/02/2025 02:17:04.858 7f6e73694048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:04.858 7f6e73694048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:17:04.859 7f6e73694048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:17:04.859 7f6e73694048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:04.859 7f6e73694048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:17:04.859 7f6e73694048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:04.859 7f6e73694048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:04.859 7f6e73694048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:17:04.860 7f6e73694048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:08.861 7f6e736a6978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:17:13.866 7f6e736a6978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:17:13.866 7f6e736a6978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:17:13.866 7f6e736a6978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:13.867 7f6e736a6978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:17:13.867 7f6e736a6978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:13.867 7f6e736a6978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:17:13.867 7f6e736a6978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:17:13.867 7f6e736a6978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:13.867 7f6e736a6978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:17:13.867 7f6e736a6978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:13.867 7f6e736a6978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:13.867 7f6e736a6978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:17:13.867 7f6e736a6978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:17.870 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:17:22.873 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:17:22.873 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:17:22.873 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:22.873 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:17:22.873 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:22.873 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:17:22.874 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:17:22.874 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:22.874 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:17:22.874 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:22.874 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:22.874 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:17:22.874 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:26.877 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:17:31.883 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:17:31.883 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:17:31.883 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:31.883 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:17:31.883 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:31.883 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:17:31.884 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:17:31.884 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:31.884 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:17:31.884 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:31.884 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:31.884 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:17:31.884 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:35.888 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:17:40.895 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:17:40.895 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:17:40.896 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:40.896 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:17:40.896 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:40.896 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:17:40.897 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:17:40.897 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:40.897 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:17:40.897 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:40.897 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:40.897 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:17:40.897 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:44.896 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:17:49.903 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:17:49.903 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:17:49.903 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:49.903 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:17:49.903 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:49.903 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:17:49.904 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:17:49.904 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:49.904 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:17:49.904 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:49.904 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:49.904 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:17:49.904 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:53.906 7f6e7369a978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:17:58.909 7f6e7369a978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:17:58.909 7f6e7369a978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:17:58.909 7f6e7369a978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:58.910 7f6e7369a978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:17:58.910 7f6e7369a978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:58.910 7f6e7369a978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:17:58.910 7f6e7369a978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:17:58.910 7f6e7369a978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:58.911 7f6e7369a978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:17:58.911 7f6e7369a978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:58.911 7f6e7369a978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:58.911 7f6e7369a978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:17:58.911 7f6e7369a978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:02.911 7f6e72e25198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:18:03.521 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51356548, partition_space_available_1k=29733936, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29476, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94116, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59384, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15213704, free=9569392, buffers=994116, cached=6887908, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.21, five_min_avg=4.64, fifteen_min_avg=4.37, cpu_share=2.62, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:18:07.913 7f6e72e25198 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:18:07.913 7f6e72e25198 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:18:07.913 7f6e72e25198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:07.913 7f6e72e25198 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:18:07.913 7f6e72e25198 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:07.913 7f6e72e25198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:07.913 7f6e72e25198 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51356548 partition_space_available_1k = 29733936 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29476 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94116 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59384 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15213704 free = 9569392 buffers = 994116 cached = 6887908 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 3.21 five_min_avg = 4.64 fifteen_min_avg = 4.37 cpu_share = 2.62 node_type = analytics-node >> >> 06/02/2025 02:18:07.914 7f6e72e25198 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:07.914 7f6e72e25198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:18:07.914 7f6e72e25198 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:18:07.914 7f6e72e25198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:07.914 7f6e72e25198 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:18:07.914 7f6e72e25198 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:07.915 7f6e72e25198 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:07.915 7f6e72e25198 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:18:07.915 7f6e72e25198 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:11.917 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:18:16.923 7f6e736a6f60 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:18:16.924 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:18:16.924 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:16.924 7f6e736a6f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:18:16.924 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:16.924 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:18:16.925 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:18:16.925 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:16.925 7f6e736a6f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:18:16.925 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:16.925 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:16.925 7f6e736a6f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:18:16.925 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:20.927 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:18:25.933 7f6e736a6f60 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:18:25.933 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:18:25.934 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:25.934 7f6e736a6f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:18:25.934 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:25.934 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:18:25.934 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:18:25.934 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:25.934 7f6e736a6f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:18:25.934 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:25.935 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:25.935 7f6e736a6f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:18:25.935 7f6e736a6f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:29.938 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:18:34.944 7f6e7369a4e0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:18:34.944 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:18:34.945 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:34.945 7f6e7369a4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:18:34.945 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:34.945 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:18:34.946 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:18:34.946 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:34.946 7f6e7369a4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:18:34.946 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:34.946 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:34.946 7f6e7369a4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:18:34.946 7f6e7369a4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:38.947 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:18:43.953 7f6e736986d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:18:43.953 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:18:43.954 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:43.954 7f6e736986d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:18:43.954 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:43.954 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:18:43.955 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:18:43.955 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:43.955 7f6e736986d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:18:43.955 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:43.955 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:43.955 7f6e736986d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:18:43.955 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:47.957 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:18:52.961 7f6e736986d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:18:52.961 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:18:52.961 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:52.961 7f6e736986d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:18:52.961 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:52.961 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:18:52.962 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:18:52.962 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:52.962 7f6e736986d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:18:52.962 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:52.962 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:52.962 7f6e736986d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:18:52.962 7f6e736986d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:56.965 7f6e722306d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:19:01.967 7f6e722306d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:19:01.968 7f6e722306d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:19:01.968 7f6e722306d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:01.969 7f6e722306d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:19:01.969 7f6e722306d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:01.969 7f6e722306d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:19:01.970 7f6e722306d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:19:01.971 7f6e722306d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:01.971 7f6e722306d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:19:01.971 7f6e722306d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:01.971 7f6e722306d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:01.972 7f6e722306d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:19:01.972 7f6e722306d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:03.521 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51362004, partition_space_available_1k=29728480, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29480, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94116, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59376, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15202108, free=9574464, buffers=994124, cached=6894424, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.39, five_min_avg=4.15, fifteen_min_avg=4.22, cpu_share=2.75, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:19:03.522 7f6e73694ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:03.522 7f6e73694ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51362004 partition_space_available_1k = 29728480 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29480 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94116 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59376 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15202108 free = 9574464 buffers = 994124 cached = 6894424 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 2.39 five_min_avg = 4.15 fifteen_min_avg = 4.22 cpu_share = 2.75 node_type = analytics-node >> >> 06/02/2025 02:19:03.522 7f6e73694ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:05.969 7f6e771dd828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:19:10.974 7f6e771dd828 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:19:10.975 7f6e771dd828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:19:10.975 7f6e771dd828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:10.975 7f6e771dd828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:19:10.975 7f6e771dd828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:10.975 7f6e771dd828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:19:10.975 7f6e771dd828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:19:10.976 7f6e771dd828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:10.976 7f6e771dd828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:19:10.976 7f6e771dd828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:10.976 7f6e771dd828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:10.976 7f6e771dd828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:19:10.976 7f6e771dd828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:14.978 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:19:19.984 7f6e72245588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:19:19.984 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:19:19.984 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:19.985 7f6e72245588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:19:19.985 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:19.985 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:19:19.985 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:19:19.986 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:19.986 7f6e72245588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:19:19.986 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:19.986 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:19.986 7f6e72245588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:19:19.986 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:23.988 7f6e771ddf60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:19:28.994 7f6e771ddf60 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:19:28.995 7f6e771ddf60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:19:28.995 7f6e771ddf60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:28.995 7f6e771ddf60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:19:28.995 7f6e771ddf60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:28.995 7f6e771ddf60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:19:28.996 7f6e771ddf60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:19:28.996 7f6e771ddf60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:28.996 7f6e771ddf60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:19:28.996 7f6e771ddf60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:28.996 7f6e771ddf60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:28.996 7f6e771ddf60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:19:28.996 7f6e771ddf60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:32.999 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:19:38.001 7f6e72245588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:19:38.002 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:19:38.002 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:38.002 7f6e72245588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:19:38.002 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:38.002 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:19:38.002 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:19:38.002 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:38.002 7f6e72245588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:19:38.003 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:38.003 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:38.003 7f6e72245588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:19:38.003 7f6e72245588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:42.005 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:19:47.009 7f6e72e28828 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:19:47.009 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:19:47.009 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:47.009 7f6e72e28828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:19:47.009 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:47.009 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:19:47.010 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:19:47.010 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:47.010 7f6e72e28828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:19:47.010 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:47.010 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:47.010 7f6e72e28828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:19:47.010 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:51.014 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:19:56.019 7f6e72e28828 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:19:56.019 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:19:56.019 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:56.020 7f6e72e28828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:19:56.020 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:56.020 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:19:56.020 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:19:56.021 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:56.021 7f6e72e28828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:19:56.021 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:56.021 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:56.021 7f6e72e28828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:19:56.021 7f6e72e28828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:00.020 7f6e77208240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:20:03.531 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51367380, partition_space_available_1k=29723104, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29480, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94148, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59432, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15245568, free=9524512, buffers=994124, cached=6900916, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.26, five_min_avg=3.81, fifteen_min_avg=4.1, cpu_share=2.75, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:20:05.024 7f6e77208240 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:20:05.025 7f6e77208240 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:20:05.025 7f6e77208240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:05.025 7f6e77208240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:20:05.025 7f6e77208240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:05.025 7f6e77208240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:05.026 7f6e77208240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51367380 partition_space_available_1k = 29723104 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29480 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94148 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59432 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15245568 free = 9524512 buffers = 994124 cached = 6900916 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 2.26 five_min_avg = 3.81 fifteen_min_avg = 4.1 cpu_share = 2.75 node_type = analytics-node >> >> 06/02/2025 02:20:05.026 7f6e77208240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:05.026 7f6e77208240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:20:05.026 7f6e77208240 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:20:05.026 7f6e77208240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:05.027 7f6e77208240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:20:05.027 7f6e77208240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:05.027 7f6e77208240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:05.027 7f6e77208240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:20:05.027 7f6e77208240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:09.027 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:20:14.034 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:20:14.035 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:20:14.035 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:14.035 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:20:14.035 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:14.035 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:20:14.036 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:20:14.036 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:14.036 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:20:14.036 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:14.037 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:14.037 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:20:14.037 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:18.037 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:20:22.549 7f6e771eacc0 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:20:23.041 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:20:23.042 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:20:23.042 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:23.042 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:20:23.042 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:23.042 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:20:23.043 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:20:23.043 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:23.043 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:20:23.043 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:23.043 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:23.043 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:20:23.043 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:27.045 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:20:32.049 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:20:32.049 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:20:32.049 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:32.049 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:20:32.049 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:32.049 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:20:32.050 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:20:32.050 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:32.050 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:20:32.050 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:32.050 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:32.050 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:20:32.050 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:33.253 7f6e75787588 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:20:36.051 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:20:41.057 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:20:41.057 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:20:41.057 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:41.058 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:20:41.058 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:41.058 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:20:41.058 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:20:41.058 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:41.058 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:20:41.058 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:41.058 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:41.059 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:20:41.059 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:43.677 7f6e75787588 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:20:45.059 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:20:50.066 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:20:50.066 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:20:50.066 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:50.066 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:20:50.066 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:50.067 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:20:50.067 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:20:50.067 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:50.068 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:20:50.068 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:50.068 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:50.068 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:20:50.068 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:54.067 7f6e771ea240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:20:59.069 7f6e771ea240 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:20:59.070 7f6e771ea240 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:20:59.070 7f6e771ea240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:59.070 7f6e771ea240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:20:59.070 7f6e771ea240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:59.070 7f6e771ea240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:20:59.071 7f6e771ea240 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:20:59.071 7f6e771ea240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:59.071 7f6e771ea240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:20:59.071 7f6e771ea240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:59.072 7f6e771ea240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:59.072 7f6e771ea240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:20:59.072 7f6e771ea240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:03.071 7f6e771dd978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:21:03.541 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51376960, partition_space_available_1k=29713524, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29484, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94208, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59452, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15235636, free=9520648, buffers=995508, cached=6913328, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.16, five_min_avg=3.51, fifteen_min_avg=3.98, cpu_share=3.12, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:21:08.074 7f6e771dd978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:21:08.074 7f6e771dd978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:21:08.074 7f6e771dd978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:08.074 7f6e771dd978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:21:08.074 7f6e771dd978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:08.074 7f6e771dd978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:08.075 7f6e771dd978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51376960 partition_space_available_1k = 29713524 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29484 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94208 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59452 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15235636 free = 9520648 buffers = 995508 cached = 6913328 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 2.16 five_min_avg = 3.51 fifteen_min_avg = 3.98 cpu_share = 3.12 node_type = analytics-node >> >> 06/02/2025 02:21:08.075 7f6e771dd978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:08.075 7f6e771dd978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:21:08.075 7f6e771dd978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:21:08.075 7f6e771dd978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:08.076 7f6e771dd978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:21:08.076 7f6e771dd978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:08.076 7f6e771dd978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:08.076 7f6e771dd978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:21:08.076 7f6e771dd978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:12.077 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:21:17.082 7f6e736a70f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:21:17.082 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:21:17.082 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:17.083 7f6e736a70f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:21:17.083 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:17.083 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:21:17.083 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:21:17.083 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:17.083 7f6e736a70f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:21:17.083 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:17.084 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:17.084 7f6e736a70f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:21:17.084 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:21.086 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:21:26.092 7f6e736a70f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:21:26.092 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:21:26.092 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:26.092 7f6e736a70f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:21:26.093 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:26.093 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:21:26.093 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:21:26.093 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:26.093 7f6e736a70f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:21:26.093 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:26.093 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:26.094 7f6e736a70f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:21:26.094 7f6e736a70f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:30.097 7f6e75d04eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:21:35.101 7f6e75d04eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:21:35.101 7f6e75d04eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:21:35.101 7f6e75d04eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:35.102 7f6e75d04eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:21:35.102 7f6e75d04eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:35.102 7f6e75d04eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:21:35.102 7f6e75d04eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:21:35.102 7f6e75d04eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:35.102 7f6e75d04eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:21:35.102 7f6e75d04eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:35.102 7f6e75d04eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:35.102 7f6e75d04eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:21:35.102 7f6e75d04eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:39.102 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:21:44.109 7f6e771ddcc0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:21:44.109 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:21:44.111 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:44.111 7f6e771ddcc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:21:44.111 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:44.113 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:21:44.114 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:21:44.114 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:44.114 7f6e771ddcc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:21:44.114 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:44.114 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:44.114 7f6e771ddcc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:21:44.114 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:48.112 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:21:52.246 7f6e7369a780 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:21:53.115 7f6e771ddcc0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:21:53.115 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:21:53.115 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:53.115 7f6e771ddcc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:21:53.115 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:53.115 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:21:53.116 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:21:53.116 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:53.116 7f6e771ddcc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:21:53.116 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:53.116 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:53.116 7f6e771ddcc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:21:53.116 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:57.118 7f6e75cf2cc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:22:02.120 7f6e75cf2cc0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:22:02.121 7f6e75cf2cc0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:22:02.121 7f6e75cf2cc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:02.121 7f6e75cf2cc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:22:02.121 7f6e75cf2cc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:02.121 7f6e75cf2cc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:22:02.122 7f6e75cf2cc0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:22:02.122 7f6e75cf2cc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:02.122 7f6e75cf2cc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:22:02.122 7f6e75cf2cc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:02.122 7f6e75cf2cc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:02.122 7f6e75cf2cc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:22:02.122 7f6e75cf2cc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:03.530 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51384352, partition_space_available_1k=29706132, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29484, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94204, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59444, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15283752, free=9454252, buffers=996092, cached=6931024, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.59, five_min_avg=4.37, fifteen_min_avg=4.25, cpu_share=5.12, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:22:03.531 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:03.531 7f6e75b14780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51384352 partition_space_available_1k = 29706132 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29484 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94204 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59444 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15283752 free = 9454252 buffers = 996092 cached = 6931024 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 5.59 five_min_avg = 4.37 fifteen_min_avg = 4.25 cpu_share = 5.12 node_type = analytics-node >> >> 06/02/2025 02:22:03.531 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:06.123 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:22:11.126 7f6e75b14780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:22:11.126 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:22:11.126 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:11.127 7f6e75b14780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:22:11.127 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:11.127 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:22:11.127 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:22:11.127 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:11.127 7f6e75b14780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:22:11.127 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:11.127 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:11.127 7f6e75b14780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:22:11.127 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:15.128 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:22:20.134 7f6e75b14780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:22:20.135 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:22:20.135 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:20.135 7f6e75b14780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:22:20.135 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:20.135 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:22:20.135 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:22:20.136 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:20.136 7f6e75b14780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:22:20.136 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:20.136 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:20.136 7f6e75b14780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:22:20.136 7f6e75b14780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:24.135 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:22:29.141 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:22:29.141 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:22:29.141 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:29.141 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:22:29.141 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:29.141 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:22:29.142 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:22:29.142 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:29.142 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:22:29.142 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:29.142 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:29.142 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:22:29.142 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:33.145 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:22:38.152 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:22:38.152 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:22:38.152 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:38.153 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:22:38.153 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:38.153 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:22:38.153 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:22:38.153 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:38.153 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:22:38.154 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:38.154 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:38.154 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:22:38.154 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:42.152 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:22:47.158 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:22:47.159 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:22:47.159 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:47.159 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:22:47.159 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:47.159 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:22:47.159 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:22:47.160 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:47.160 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:22:47.160 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:47.160 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:47.160 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:22:47.160 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:51.163 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:22:56.167 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:22:56.167 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:22:56.167 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:56.167 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:22:56.167 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:56.167 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:22:56.168 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:22:56.168 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:56.168 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:22:56.168 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:56.168 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:56.168 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:22:56.168 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:00.168 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:23:03.531 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51392008, partition_space_available_1k=29698476, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29484, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94240, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59500, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15268044, free=9460932, buffers=996100, cached=6940044, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.56, five_min_avg=4.03, fifteen_min_avg=4.14, cpu_share=2.38, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:23:05.171 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:23:05.172 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:23:05.172 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:05.172 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:23:05.172 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:05.172 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:05.173 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51392008 partition_space_available_1k = 29698476 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29484 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94240 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59500 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15268044 free = 9460932 buffers = 996100 cached = 6940044 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 3.56 five_min_avg = 4.03 fifteen_min_avg = 4.14 cpu_share = 2.38 node_type = analytics-node >> >> 06/02/2025 02:23:05.173 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:05.173 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:23:05.173 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:23:05.173 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:05.173 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:23:05.173 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:05.173 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:05.173 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:23:05.173 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:09.173 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:23:14.179 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:23:14.180 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:23:14.181 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:14.181 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:23:14.181 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:14.181 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:23:14.181 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:23:14.182 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:14.182 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:23:14.182 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:14.182 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:14.182 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:23:14.182 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:18.182 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:23:23.189 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:23:23.189 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:23:23.190 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:23.190 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:23:23.190 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:23.190 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:23:23.191 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:23:23.191 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:23.191 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:23:23.191 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:23.191 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:23.191 7f6e7222bc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:23:23.191 7f6e7222bc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:27.192 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:23:32.197 7f6e72e250f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:23:32.197 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:23:32.197 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:32.197 7f6e72e250f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:23:32.197 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:32.197 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:23:32.198 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:23:32.198 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:32.198 7f6e72e250f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:23:32.198 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:32.198 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:32.198 7f6e72e250f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:23:32.198 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:36.201 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:23:41.204 7f6e72e250f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:23:41.204 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:23:41.204 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:41.204 7f6e72e250f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:23:41.204 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:41.205 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:23:41.205 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:23:41.205 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:41.205 7f6e72e250f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:23:41.205 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:41.205 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:41.205 7f6e72e250f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:23:41.205 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:45.206 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:23:50.213 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:23:50.213 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:23:50.213 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:50.213 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:23:50.213 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:50.213 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:23:50.213 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:23:50.214 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:50.214 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:23:50.214 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:50.214 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:50.214 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:23:50.214 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:54.217 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:23:59.220 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:23:59.220 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:23:59.221 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:59.221 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:23:59.221 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:59.221 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:23:59.222 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:23:59.222 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:59.222 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:23:59.222 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:59.222 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:59.222 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:23:59.222 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:03.222 7f6e771ea780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:24:03.537 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51392464, partition_space_available_1k=29698020, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29484, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94292, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59520, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15264824, free=9463096, buffers=996100, cached=6941100, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.96, five_min_avg=3.75, fifteen_min_avg=4.04, cpu_share=2.25, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:24:08.229 7f6e771ea780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:24:08.229 7f6e771ea780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:24:08.229 7f6e771ea780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:08.230 7f6e771ea780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:24:08.230 7f6e771ea780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:08.230 7f6e771ea780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:08.230 7f6e771ea780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51392464 partition_space_available_1k = 29698020 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29484 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94292 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59520 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15264824 free = 9463096 buffers = 996100 cached = 6941100 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 2.96 five_min_avg = 3.75 fifteen_min_avg = 4.04 cpu_share = 2.25 node_type = analytics-node >> >> 06/02/2025 02:24:08.230 7f6e771ea780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:08.230 7f6e771ea780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:24:08.231 7f6e771ea780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:24:08.231 7f6e771ea780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:08.231 7f6e771ea780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:24:08.231 7f6e771ea780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:08.231 7f6e771ea780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:08.231 7f6e771ea780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:24:08.231 7f6e771ea780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:12.231 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:24:17.238 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:24:17.238 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:24:17.238 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:17.239 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:24:17.239 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:17.239 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:24:17.239 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:24:17.239 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:17.239 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:24:17.240 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:17.240 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:17.240 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:24:17.240 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:21.242 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:24:26.245 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:24:26.245 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:24:26.245 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:26.245 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:24:26.245 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:26.245 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:24:26.246 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:24:26.246 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:26.246 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:24:26.247 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:26.247 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:26.247 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:24:26.247 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:30.249 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:24:35.253 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:24:35.253 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:24:35.253 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:35.253 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:24:35.253 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:35.253 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:24:35.254 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:24:35.254 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:35.254 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:24:35.254 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:35.254 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:35.254 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:24:35.254 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:39.257 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:24:44.263 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:24:44.263 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:24:44.263 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:44.263 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:24:44.264 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:44.264 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:24:44.264 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:24:44.264 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:44.264 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:24:44.264 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:44.264 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:44.264 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:24:44.264 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:48.265 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:24:53.270 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:24:53.270 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:24:53.270 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:53.271 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:24:53.271 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:53.271 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:24:53.271 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:24:53.272 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:53.272 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:24:53.272 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:53.272 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:53.272 7f6e72230978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:24:53.272 7f6e72230978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:57.271 7f6e756410f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:25:02.273 7f6e756410f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:25:02.273 7f6e756410f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:25:02.273 7f6e756410f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:02.273 7f6e756410f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:25:02.273 7f6e756410f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:02.273 7f6e756410f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:25:02.274 7f6e756410f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:25:02.274 7f6e756410f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:02.274 7f6e756410f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:25:02.274 7f6e756410f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:02.275 7f6e756410f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:02.275 7f6e756410f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:25:02.275 7f6e756410f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:03.541 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51397840, partition_space_available_1k=29692644, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29484, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94292, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59516, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15292252, free=9433700, buffers=996100, cached=6943068, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.87, five_min_avg=3.31, fifteen_min_avg=3.87, cpu_share=2.5, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:25:03.541 7f6e736a6828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:03.542 7f6e736a6828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51397840 partition_space_available_1k = 29692644 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29484 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94292 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59516 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15292252 free = 9433700 buffers = 996100 cached = 6943068 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 1.87 five_min_avg = 3.31 fifteen_min_avg = 3.87 cpu_share = 2.5 node_type = analytics-node >> >> 06/02/2025 02:25:03.542 7f6e736a6828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:06.276 7f6e73694c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:25:11.279 7f6e73694c18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:25:11.279 7f6e73694c18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:25:11.279 7f6e73694c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:11.279 7f6e73694c18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:25:11.279 7f6e73694c18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:11.279 7f6e73694c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:25:11.280 7f6e73694c18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:25:11.280 7f6e73694c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:11.280 7f6e73694c18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:25:11.280 7f6e73694c18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:11.281 7f6e73694c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:11.281 7f6e73694c18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:25:11.281 7f6e73694c18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:15.280 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:25:20.287 7f6e73a776d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:25:20.287 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:25:20.287 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:20.287 7f6e73a776d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:25:20.287 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:20.287 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:25:20.288 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:25:20.288 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:20.288 7f6e73a776d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:25:20.288 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:20.288 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:20.288 7f6e73a776d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:25:20.288 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:24.289 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:25:29.293 7f6e73a776d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:25:29.293 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:25:29.293 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:29.294 7f6e73a776d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:25:29.294 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:29.294 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:25:29.294 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:25:29.294 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:29.295 7f6e73a776d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:25:29.295 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:29.295 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:29.295 7f6e73a776d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:25:29.295 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:33.297 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:25:38.309 7f6e73a776d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:25:38.309 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:25:38.309 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:38.310 7f6e73a776d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:25:38.310 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:38.314 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:25:38.315 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:25:38.315 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:38.315 7f6e73a776d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:25:38.315 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:38.315 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:38.317 7f6e73a776d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:25:38.317 7f6e73a776d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:42.311 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:25:47.317 7f6e7369a588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:25:47.317 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:25:47.317 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:47.317 7f6e7369a588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:25:47.317 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:47.317 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:25:47.318 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:25:47.318 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:47.318 7f6e7369a588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:25:47.318 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:47.318 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:47.318 7f6e7369a588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:25:47.318 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:51.321 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:25:56.323 7f6e7369a588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:25:56.323 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:25:56.323 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:56.323 7f6e7369a588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:25:56.323 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:56.323 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:25:56.324 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:25:56.324 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:56.324 7f6e7369a588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:25:56.324 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:56.324 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:56.324 7f6e7369a588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:25:56.324 7f6e7369a588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:00.325 7f6e736a6b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:26:03.560 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51506680, partition_space_available_1k=29583804, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29484, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94324, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59568, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15312892, free=9392292, buffers=996100, cached=6963836, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.24, five_min_avg=3.93, fifteen_min_avg=4.04, cpu_share=4.88, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:26:05.329 7f6e736a6b70 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:26:05.329 7f6e736a6b70 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:26:05.329 7f6e736a6b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:05.329 7f6e736a6b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:26:05.329 7f6e736a6b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:05.330 7f6e736a6b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:05.330 7f6e736a6b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51506680 partition_space_available_1k = 29583804 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29484 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94324 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59568 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15312892 free = 9392292 buffers = 996100 cached = 6963836 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 5.24 five_min_avg = 3.93 fifteen_min_avg = 4.04 cpu_share = 4.88 node_type = analytics-node >> >> 06/02/2025 02:26:05.330 7f6e736a6b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:05.330 7f6e736a6b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:26:05.330 7f6e736a6b70 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:26:05.331 7f6e736a6b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:05.331 7f6e736a6b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:26:05.331 7f6e736a6b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:05.331 7f6e736a6b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:05.331 7f6e736a6b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:26:05.331 7f6e736a6b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:09.333 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:26:14.338 7f6e771fbd68 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:26:14.338 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:26:14.338 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:14.338 7f6e771fbd68 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:26:14.338 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:14.338 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:26:14.339 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:26:14.339 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:14.339 7f6e771fbd68 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:26:14.339 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:14.339 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:14.339 7f6e771fbd68 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:26:14.339 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:14.501 7f6e771fbd68 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:26:18.342 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:26:23.348 7f6e771fbd68 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:26:23.348 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:26:23.348 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:23.349 7f6e771fbd68 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:26:23.349 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:23.349 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:26:23.349 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:26:23.349 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:23.349 7f6e771fbd68 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:26:23.349 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:23.349 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:23.350 7f6e771fbd68 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:26:23.350 7f6e771fbd68 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:24.802 7f6e771fbd68 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:26:27.349 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:26:32.354 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:26:32.354 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:26:32.354 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:32.354 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:26:32.354 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:32.354 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:26:32.355 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:26:32.355 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:32.355 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:26:32.355 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:32.355 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:32.355 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:26:32.355 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:35.333 7f6e759e30f0 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:26:36.355 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:26:41.362 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:26:41.362 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:26:41.362 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:41.362 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:26:41.362 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:41.363 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:26:41.363 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:26:41.363 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:41.363 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:26:41.363 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:41.363 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:41.363 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:26:41.363 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:45.364 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:26:50.369 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:26:50.369 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:26:50.369 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:50.369 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:26:50.369 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:50.369 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:26:50.370 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:26:50.370 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:50.370 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:26:50.370 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:50.370 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:50.370 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:26:50.370 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:54.373 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:26:59.375 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:26:59.376 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:26:59.376 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:59.376 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:26:59.376 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:59.376 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:26:59.377 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:26:59.378 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:59.378 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:26:59.378 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:59.378 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:59.378 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:26:59.378 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:03.377 7f6e7369ae10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:27:03.541 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51410504, partition_space_available_1k=29679980, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29488, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94388, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59592, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15305940, free=9482492, buffers=996144, cached=6880544, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.62, five_min_avg=3.78, fifteen_min_avg=3.99, cpu_share=2.88, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:27:08.382 7f6e7369ae10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:27:08.382 7f6e7369ae10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:27:08.383 7f6e7369ae10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:08.383 7f6e7369ae10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:27:08.383 7f6e7369ae10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:08.383 7f6e7369ae10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:08.383 7f6e7369ae10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51410504 partition_space_available_1k = 29679980 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29488 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94388 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59592 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15305940 free = 9482492 buffers = 996144 cached = 6880544 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 3.62 five_min_avg = 3.78 fifteen_min_avg = 3.99 cpu_share = 2.88 node_type = analytics-node >> >> 06/02/2025 02:27:08.383 7f6e7369ae10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:08.383 7f6e7369ae10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:27:08.384 7f6e7369ae10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:27:08.384 7f6e7369ae10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:08.384 7f6e7369ae10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:27:08.384 7f6e7369ae10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:08.384 7f6e7369ae10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:08.384 7f6e7369ae10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:27:08.384 7f6e7369ae10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:12.383 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:27:17.388 7f6e736944e0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:27:17.389 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:27:17.389 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:17.389 7f6e736944e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:27:17.389 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:17.389 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:27:17.390 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:27:17.390 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:17.390 7f6e736944e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:27:17.390 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:17.390 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:17.390 7f6e736944e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:27:17.390 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:21.393 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:27:26.398 7f6e736944e0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:27:26.398 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:27:26.398 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:26.398 7f6e736944e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:27:26.398 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:26.399 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:27:26.399 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:27:26.399 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:26.399 7f6e736944e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:27:26.399 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:26.399 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:26.399 7f6e736944e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:27:26.399 7f6e736944e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:30.401 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:27:35.407 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:27:35.408 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:27:35.408 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:35.408 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:27:35.408 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:35.408 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:27:35.409 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:27:35.409 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:35.409 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:27:35.409 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:35.409 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:35.409 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:27:35.409 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:39.411 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:27:39.625 7f6e7369a828 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:27:44.416 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:27:44.416 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:27:44.416 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:44.417 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:27:44.417 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:44.417 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:27:44.417 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:27:44.417 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:44.417 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:27:44.418 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:44.418 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:44.418 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:27:44.418 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:48.419 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:27:53.425 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:27:53.425 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:27:53.425 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:53.425 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:27:53.425 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:53.425 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:27:53.426 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:27:53.426 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:53.426 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:27:53.426 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:53.426 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:53.426 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:27:53.426 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:57.425 7f6e72e28eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:28:02.429 7f6e72e28eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:28:02.429 7f6e72e28eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:28:02.429 7f6e72e28eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:02.431 7f6e72e28eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:28:02.431 7f6e72e28eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:02.431 7f6e72e28eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:28:02.431 7f6e72e28eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:28:02.431 7f6e72e28eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:02.431 7f6e72e28eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:28:02.432 7f6e72e28eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:02.432 7f6e72e28eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:02.432 7f6e72e28eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:28:02.432 7f6e72e28eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:03.542 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51412768, partition_space_available_1k=29677716, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29488, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94392, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59584, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15292956, free=9492172, buffers=996164, cached=6883828, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.47, five_min_avg=3.69, fifteen_min_avg=3.94, cpu_share=2.62, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:28:03.543 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:03.543 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51412768 partition_space_available_1k = 29677716 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29488 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94392 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59584 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15292956 free = 9492172 buffers = 996164 cached = 6883828 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 3.47 five_min_avg = 3.69 fifteen_min_avg = 3.94 cpu_share = 2.62 node_type = analytics-node >> >> 06/02/2025 02:28:03.543 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:06.431 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:28:11.438 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:28:11.438 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:28:11.438 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:11.438 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:28:11.439 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:11.439 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:28:11.439 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:28:11.439 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:11.440 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:28:11.440 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:11.440 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:11.440 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:28:11.440 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:15.441 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:28:20.448 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:28:20.448 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:28:20.448 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:20.449 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:28:20.449 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:20.449 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:28:20.449 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:28:20.449 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:20.449 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:28:20.449 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:20.449 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:20.449 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:28:20.449 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:24.448 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:28:29.452 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:28:29.452 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:28:29.452 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:29.453 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:28:29.453 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:29.453 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:28:29.453 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:28:29.453 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:29.453 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:28:29.453 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:29.453 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:29.453 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:28:29.453 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:33.457 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:28:38.463 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:28:38.463 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:28:38.463 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:38.464 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:28:38.464 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:38.464 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:28:38.464 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:28:38.464 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:38.465 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:28:38.465 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:38.465 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:38.465 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:28:38.465 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:42.468 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:28:47.473 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:28:47.473 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:28:47.473 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:47.473 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:28:47.473 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:47.473 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:28:47.474 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:28:47.474 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:47.474 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:28:47.474 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:47.474 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:47.475 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:28:47.475 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:51.477 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:28:56.483 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:28:56.483 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:28:56.483 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:56.483 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:28:56.483 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:56.483 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:28:56.484 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:28:56.484 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:56.484 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:28:56.484 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:56.484 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:56.484 7f6e7c848eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:28:56.484 7f6e7c848eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:00.485 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:29:03.539 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51413364, partition_space_available_1k=29677120, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29488, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94420, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59640, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15288704, free=9494988, buffers=996172, cached=6885256, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.54, five_min_avg=3.36, fifteen_min_avg=3.81, cpu_share=2.62, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:29:05.489 7f6e771ddcc0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:29:05.489 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:29:05.489 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:05.489 7f6e771ddcc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:29:05.489 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:05.489 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:05.489 7f6e771ddcc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51413364 partition_space_available_1k = 29677120 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29488 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94420 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59640 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15288704 free = 9494988 buffers = 996172 cached = 6885256 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 2.54 five_min_avg = 3.36 fifteen_min_avg = 3.81 cpu_share = 2.62 node_type = analytics-node >> >> 06/02/2025 02:29:05.489 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:05.489 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:29:05.490 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:29:05.490 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:05.490 7f6e771ddcc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:29:05.490 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:05.490 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:05.490 7f6e771ddcc0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:29:05.490 7f6e771ddcc0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:09.492 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:29:14.496 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:29:14.496 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:29:14.496 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:14.496 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:29:14.496 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:14.496 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:29:14.497 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:29:14.497 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:14.497 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:29:14.497 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:14.497 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:14.497 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:29:14.497 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:18.499 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:29:23.505 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:29:23.505 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:29:23.505 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:23.505 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:29:23.505 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:23.506 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:29:23.506 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:29:23.506 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:23.506 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:29:23.506 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:23.506 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:23.506 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:29:23.506 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:27.509 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:29:32.515 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:29:32.515 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:29:32.515 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:32.516 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:29:32.516 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:32.516 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:29:32.516 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:29:32.516 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:32.516 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:29:32.516 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:32.516 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:32.517 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:29:32.517 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:36.520 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:29:41.526 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:29:41.526 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:29:41.526 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:41.527 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:29:41.527 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:41.527 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:29:41.527 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:29:41.527 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:41.527 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:29:41.528 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:41.528 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:41.528 7f6e7222b6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:29:41.528 7f6e7222b6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:45.529 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:29:50.535 7f6e7369a048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:29:50.535 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:29:50.535 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:50.536 7f6e7369a048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:29:50.536 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:50.536 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:29:50.536 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:29:50.536 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:50.536 7f6e7369a048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:29:50.536 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:50.536 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:50.537 7f6e7369a048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:29:50.537 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:54.539 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:29:59.541 7f6e7369a048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:29:59.542 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:29:59.542 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:59.542 7f6e7369a048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:29:59.542 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:59.542 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:29:59.543 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:29:59.543 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:59.543 7f6e7369a048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:29:59.543 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:59.543 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:59.544 7f6e7369a048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:29:59.544 7f6e7369a048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:03.543 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51418728, partition_space_available_1k=29671756, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29488, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94480, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59636, cpu_share=0.01, mem_res=39604, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15330944, free=9443332, buffers=996572, cached=6894272, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.23, five_min_avg=3.12, fifteen_min_avg=3.7, cpu_share=2.5, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:30:03.543 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:03.543 7f6e77200b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51418728 partition_space_available_1k = 29671756 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29488 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94480 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59636 cpu_share = 0.01 mem_res = 39604 >> } system_mem_usage = << total = 32665120 used = 15330944 free = 9443332 buffers = 996572 cached = 6894272 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 2.23 five_min_avg = 3.12 fifteen_min_avg = 3.7 cpu_share = 2.5 node_type = analytics-node >> >> 06/02/2025 02:30:03.543 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:03.544 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:30:08.550 7f6e77200b70 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:30:08.550 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:30:08.550 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:08.551 7f6e77200b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:30:08.551 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:08.551 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:30:08.551 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:30:08.551 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:08.551 7f6e77200b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:30:08.551 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:08.551 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:08.552 7f6e77200b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:30:08.552 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:12.553 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:30:17.559 7f6e77200b70 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:30:17.560 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:30:17.560 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:17.560 7f6e77200b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:30:17.560 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:17.563 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:30:17.564 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:30:17.564 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:17.564 7f6e77200b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:30:17.564 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:17.564 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:17.564 7f6e77200b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:30:17.565 7f6e77200b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:21.560 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:30:26.564 7f6e72e250f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:30:26.565 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:30:26.565 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:26.565 7f6e72e250f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:30:26.565 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:26.565 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:30:26.566 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:30:26.566 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:26.566 7f6e72e250f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:30:26.566 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:26.566 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:26.566 7f6e72e250f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:30:26.566 7f6e72e250f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:30.566 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:30:35.572 7f6e72e25588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:30:35.573 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:30:35.573 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:35.573 7f6e72e25588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:30:35.573 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:35.573 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:30:35.574 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:30:35.574 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:35.574 7f6e72e25588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:30:35.574 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:35.574 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:35.574 7f6e72e25588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:30:35.574 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:39.575 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:30:44.578 7f6e72e25588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:30:44.609 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:30:44.609 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:44.609 7f6e72e25588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:30:44.609 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:44.609 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:30:44.610 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:30:44.610 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:44.610 7f6e72e25588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:30:44.610 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:44.615 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:44.615 7f6e72e25588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:30:44.615 7f6e72e25588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:48.583 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:30:53.589 7f6e736a64e0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:30:53.590 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:30:53.590 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:53.590 7f6e736a64e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:30:53.590 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:53.590 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:30:53.591 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:30:53.591 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:53.591 7f6e736a64e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:30:53.591 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:53.591 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:53.591 7f6e736a64e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:30:53.591 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:57.593 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:30:58.286 7f6e771dd630 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:31:02.596 7f6e736a64e0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:31:02.596 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:31:02.596 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:02.596 7f6e736a64e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:31:02.596 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:02.597 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:31:02.597 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:31:02.597 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:02.597 7f6e736a64e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:31:02.598 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:02.598 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:02.598 7f6e736a64e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:31:02.598 7f6e736a64e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:03.555 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51431140, partition_space_available_1k=29659344, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29492, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94476, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59684, cpu_share=0.01, mem_res=39632, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15392536, free=9341504, buffers=997792, cached=6933288, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.38, five_min_avg=3.89, fifteen_min_avg=3.92, cpu_share=5.0, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:31:03.555 7f6e771f4f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:03.555 7f6e771f4f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51431140 partition_space_available_1k = 29659344 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29492 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94476 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59684 cpu_share = 0.01 mem_res = 39632 >> } system_mem_usage = << total = 32665120 used = 15392536 free = 9341504 buffers = 997792 cached = 6933288 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 5.38 five_min_avg = 3.89 fifteen_min_avg = 3.92 cpu_share = 5.0 node_type = analytics-node >> >> 06/02/2025 02:31:03.556 7f6e771f4f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:06.598 7f6e771f4f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:31:08.683 7f6e771ddf60 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:31:11.602 7f6e771f4f60 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:31:11.602 7f6e771f4f60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:31:11.602 7f6e771f4f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:11.603 7f6e771f4f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:31:11.603 7f6e771f4f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:11.603 7f6e771f4f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:31:11.603 7f6e771f4f60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:31:11.603 7f6e771f4f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:11.603 7f6e771f4f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:31:11.603 7f6e771f4f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:11.603 7f6e771f4f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:11.603 7f6e771f4f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:31:11.603 7f6e771f4f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:15.604 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:31:19.265 7f6e771fb438 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:31:20.606 7f6e72245c18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:31:20.606 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:31:20.606 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:20.606 7f6e72245c18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:31:20.606 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:20.606 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:31:20.607 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:31:20.607 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:20.607 7f6e72245c18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:31:20.607 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:20.607 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:20.607 7f6e72245c18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:31:20.607 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:24.609 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:31:29.610 7f6e72245c18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:31:29.611 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:31:29.611 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:29.611 7f6e72245c18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:31:29.611 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:29.611 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:31:29.612 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:31:29.612 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:29.612 7f6e72245c18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:31:29.612 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:29.612 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:29.612 7f6e72245c18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:31:29.612 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:33.615 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:31:38.621 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:31:38.621 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:31:38.622 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:38.622 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:31:38.622 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:38.622 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:31:38.623 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:31:38.623 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:38.623 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:31:38.623 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:38.623 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:38.623 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:31:38.623 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:42.625 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:31:47.630 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:31:47.630 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:31:47.630 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:47.630 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:31:47.630 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:47.630 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:31:47.631 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:31:47.631 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:47.631 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:31:47.631 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:47.631 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:47.631 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:31:47.632 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:51.633 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:31:56.639 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:31:56.639 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:31:56.640 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:56.640 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:31:56.640 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:56.640 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:31:56.641 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:31:56.641 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:56.641 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:31:56.641 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:56.641 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:56.641 7f6e7222b390 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:31:56.641 7f6e7222b390 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:00.640 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:32:03.543 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51430240, partition_space_available_1k=29660244, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29492, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94508, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59740, cpu_share=0.01, mem_res=39632, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15321820, free=9412056, buffers=997792, cached=6933452, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.21, five_min_avg=3.58, fifteen_min_avg=3.81, cpu_share=2.75, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:32:05.645 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:32:05.645 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:32:05.645 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:05.645 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:32:05.645 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:05.645 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:05.646 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51430240 partition_space_available_1k = 29660244 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29492 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94508 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59740 cpu_share = 0.01 mem_res = 39632 >> } system_mem_usage = << total = 32665120 used = 15321820 free = 9412056 buffers = 997792 cached = 6933452 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 3.21 five_min_avg = 3.58 fifteen_min_avg = 3.81 cpu_share = 2.75 node_type = analytics-node >> >> 06/02/2025 02:32:05.646 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:05.646 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:32:05.646 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:32:05.647 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:05.647 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:32:05.647 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:05.647 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:05.647 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:32:05.647 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:09.649 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:32:14.655 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:32:14.656 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:32:14.656 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:14.656 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:32:14.656 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:14.656 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:32:14.656 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:32:14.657 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:14.657 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:32:14.657 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:14.657 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:14.657 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:32:14.657 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:18.657 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:32:23.663 7f6e75d0c6d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:32:23.664 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:32:23.664 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:23.664 7f6e75d0c6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:32:23.664 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:23.664 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:32:23.665 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:32:23.665 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:23.665 7f6e75d0c6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:32:23.665 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:23.665 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:23.665 7f6e75d0c6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:32:23.665 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:27.664 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:32:32.669 7f6e75d0c6d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:32:32.669 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:32:32.669 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:32.670 7f6e75d0c6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:32:32.670 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:32.670 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:32:32.670 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:32:32.670 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:32.670 7f6e75d0c6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:32:32.670 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:32.670 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:32.671 7f6e75d0c6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:32:32.671 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:36.218 7f6e75d0c6d8 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:32:36.669 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:32:41.676 7f6e75d0c6d8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:32:41.676 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:32:41.676 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:41.676 7f6e75d0c6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:32:41.676 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:41.676 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:32:41.677 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:32:41.677 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:41.677 7f6e75d0c6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:32:41.677 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:41.677 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:41.677 7f6e75d0c6d8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:32:41.677 7f6e75d0c6d8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:45.676 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:32:50.682 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:32:50.682 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:32:50.682 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:50.682 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:32:50.682 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:50.682 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:32:50.683 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:32:50.683 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:50.683 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:32:50.685 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:50.685 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:50.685 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:32:50.685 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:54.684 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:32:59.686 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:32:59.686 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:32:59.686 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:59.687 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:32:59.687 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:59.687 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:32:59.687 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:32:59.688 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:59.688 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:32:59.688 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:59.688 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:59.688 7f6e771fd048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:32:59.688 7f6e771fd048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:03.554 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51432272, partition_space_available_1k=29658212, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29492, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94568, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59728, cpu_share=0.01, mem_res=39632, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15322252, free=9409088, buffers=997792, cached=6935988, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.63, five_min_avg=3.34, fifteen_min_avg=3.71, cpu_share=2.75, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:33:03.554 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:03.554 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51432272 partition_space_available_1k = 29658212 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29492 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94568 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59728 cpu_share = 0.01 mem_res = 39632 >> } system_mem_usage = << total = 32665120 used = 15322252 free = 9409088 buffers = 997792 cached = 6935988 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 2.63 five_min_avg = 3.34 fifteen_min_avg = 3.71 cpu_share = 2.75 node_type = analytics-node >> >> 06/02/2025 02:33:03.555 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:03.686 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:33:08.693 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:33:08.693 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:33:08.693 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:08.693 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:33:08.693 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:08.693 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:33:08.694 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:33:08.694 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:08.694 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:33:08.694 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:08.694 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:08.694 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:33:08.694 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:12.697 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:33:17.701 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:33:17.701 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:33:17.701 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:17.701 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:33:17.701 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:17.701 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:33:17.702 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:33:17.702 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:17.702 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:33:17.702 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:17.702 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:17.703 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:33:17.703 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:21.705 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:33:26.711 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:33:26.712 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:33:26.712 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:26.712 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:33:26.712 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:26.712 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:33:26.712 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:33:26.713 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:26.713 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:33:26.713 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:26.713 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:26.713 7f6e7bf8ce10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:33:26.713 7f6e7bf8ce10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:30.712 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:33:35.718 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:33:35.718 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:33:35.719 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:35.719 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:33:35.719 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:35.719 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:33:35.719 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:33:35.719 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:35.720 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:33:35.720 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:35.720 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:35.720 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:33:35.720 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:39.722 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:33:44.725 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:33:44.725 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:33:44.725 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:44.725 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:33:44.725 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:44.725 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:33:44.726 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:33:44.726 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:44.726 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:33:44.726 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:44.726 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:44.726 7f6e759e30f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:33:44.727 7f6e759e30f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:48.728 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:33:53.733 7f6e771f4828 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:33:53.733 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:33:53.733 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:53.733 7f6e771f4828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:33:53.733 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:53.733 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:33:53.734 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:33:53.734 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:53.734 7f6e771f4828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:33:53.734 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:53.734 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:53.734 7f6e771f4828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:33:53.734 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:57.737 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:34:02.740 7f6e771f4828 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:34:02.740 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:34:02.740 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:02.741 7f6e771f4828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:34:02.741 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:02.741 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:34:02.741 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:34:02.741 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:02.742 7f6e771f4828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:34:02.742 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:02.742 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:02.742 7f6e771f4828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:34:02.742 7f6e771f4828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:03.553 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51433044, partition_space_available_1k=29657440, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29492, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94568, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59756, cpu_share=0.01, mem_res=39632, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15313252, free=9416292, buffers=997792, cached=6937784, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.18, five_min_avg=3.08, fifteen_min_avg=3.6, cpu_share=2.75, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:34:03.554 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:03.554 7f6e72dba438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51433044 partition_space_available_1k = 29657440 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29492 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94568 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59756 cpu_share = 0.01 mem_res = 39632 >> } system_mem_usage = << total = 32665120 used = 15313252 free = 9416292 buffers = 997792 cached = 6937784 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 2.18 five_min_avg = 3.08 fifteen_min_avg = 3.6 cpu_share = 2.75 node_type = analytics-node >> >> 06/02/2025 02:34:03.554 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:06.741 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:34:11.748 7f6e72dba438 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:34:11.748 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:34:11.748 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:11.748 7f6e72dba438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:34:11.748 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:11.749 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:34:11.749 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:34:11.749 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:11.749 7f6e72dba438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:34:11.749 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:11.749 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:11.750 7f6e72dba438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:34:11.750 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:15.752 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:34:20.757 7f6e72dba438 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:34:20.757 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:34:20.757 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:20.758 7f6e72dba438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:34:20.758 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:20.758 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:34:20.758 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:34:20.758 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:20.759 7f6e72dba438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:34:20.759 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:20.759 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:20.759 7f6e72dba438 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:34:20.759 7f6e72dba438 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:24.760 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:34:29.765 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:34:29.765 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:34:29.765 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:29.766 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:34:29.766 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:29.766 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:34:29.766 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:34:29.766 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:29.766 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:34:29.766 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:29.766 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:29.767 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:34:29.767 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:33.767 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:34:38.773 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:34:38.773 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:34:38.773 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:38.773 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:34:38.773 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:38.773 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:34:38.774 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:34:38.774 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:38.775 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:34:38.775 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:38.775 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:38.775 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:34:38.775 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:42.777 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:34:47.781 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:34:47.781 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:34:47.781 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:47.782 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:34:47.782 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:47.782 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:34:47.782 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:34:47.782 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:47.782 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:34:47.783 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:47.783 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:47.783 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:34:47.783 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:51.785 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:34:56.791 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:34:56.792 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:34:56.792 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:56.792 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:34:56.792 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:56.792 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:34:56.793 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:34:56.793 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:56.793 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:34:56.793 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:56.793 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:56.793 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:34:56.794 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:00.793 7f6e736a8ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:35:03.557 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51438612, partition_space_available_1k=29651872, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29504, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94604, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59804, cpu_share=0.01, mem_res=39632, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15338140, free=9385200, buffers=997912, cached=6943868, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.74, five_min_avg=2.79, fifteen_min_avg=3.47, cpu_share=2.75, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:35:05.798 7f6e736a8ac8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:35:05.799 7f6e736a8ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:35:05.799 7f6e736a8ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:05.799 7f6e736a8ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:35:05.799 7f6e736a8ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:05.799 7f6e736a8ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:05.799 7f6e736a8ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51438612 partition_space_available_1k = 29651872 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29504 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94604 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59804 cpu_share = 0.01 mem_res = 39632 >> } system_mem_usage = << total = 32665120 used = 15338140 free = 9385200 buffers = 997912 cached = 6943868 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 1.74 five_min_avg = 2.79 fifteen_min_avg = 3.47 cpu_share = 2.75 node_type = analytics-node >> >> 06/02/2025 02:35:05.799 7f6e736a8ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:05.799 7f6e736a8ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:35:05.800 7f6e736a8ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:35:05.800 7f6e736a8ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:05.800 7f6e736a8ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:35:05.800 7f6e736a8ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:05.800 7f6e736a8ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:05.800 7f6e736a8ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:35:05.800 7f6e736a8ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:09.803 7f6e75d31b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:35:14.809 7f6e75d31b70 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:35:14.809 7f6e75d31b70 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:35:14.809 7f6e75d31b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:14.809 7f6e75d31b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:35:14.809 7f6e75d31b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:14.809 7f6e75d31b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:35:14.810 7f6e75d31b70 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:35:14.810 7f6e75d31b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:14.810 7f6e75d31b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:35:14.810 7f6e75d31b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:14.810 7f6e75d31b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:14.810 7f6e75d31b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:35:14.810 7f6e75d31b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:18.810 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:35:23.817 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:35:23.817 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:35:23.817 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:23.817 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:35:23.817 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:23.817 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:35:23.818 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:35:23.818 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:23.818 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:35:23.818 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:23.818 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:23.818 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:35:23.818 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:27.820 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:35:32.824 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:35:32.824 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:35:32.824 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:32.824 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:35:32.824 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:32.824 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:35:32.825 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:35:32.825 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:32.825 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:35:32.825 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:32.825 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:32.825 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:35:32.826 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:36.828 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:35:41.833 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:35:41.833 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:35:41.833 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:41.833 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:35:41.833 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:41.833 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:35:41.834 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:35:41.834 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:41.834 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:35:41.834 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:41.834 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:41.834 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:35:41.834 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:45.837 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:35:50.843 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:35:50.844 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:35:50.844 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:50.844 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:35:50.845 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:50.845 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:35:50.845 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:35:50.846 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:50.846 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:35:50.846 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:50.846 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:50.846 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:35:50.846 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:54.847 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:35:59.848 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:35:59.849 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:35:59.849 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:59.849 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:35:59.849 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:59.849 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:35:59.850 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:35:59.850 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:59.850 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:35:59.850 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:59.850 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:59.850 7f6e75b888d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:35:59.850 7f6e75b888d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:03.568 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51439248, partition_space_available_1k=29651236, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29496, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94660, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59800, cpu_share=0.01, mem_res=39632, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15315060, free=9406860, buffers=997912, cached=6945288, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.91, five_min_avg=2.66, fifteen_min_avg=3.38, cpu_share=2.88, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:36:03.568 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:03.569 7f6e771f4a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51439248 partition_space_available_1k = 29651236 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29496 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94660 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59800 cpu_share = 0.01 mem_res = 39632 >> } system_mem_usage = << total = 32665120 used = 15315060 free = 9406860 buffers = 997912 cached = 6945288 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 1.91 five_min_avg = 2.66 fifteen_min_avg = 3.38 cpu_share = 2.88 node_type = analytics-node >> >> 06/02/2025 02:36:03.569 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:03.849 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:36:08.853 7f6e771f4a20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:36:08.853 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:36:08.853 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:08.853 7f6e771f4a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:36:08.853 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:08.853 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:36:08.854 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:36:08.854 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:08.854 7f6e771f4a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:36:08.854 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:08.854 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:08.854 7f6e771f4a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:36:08.854 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:12.857 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:36:14.565 7f6e73694eb8 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:36:17.859 7f6e771f4a20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:36:17.860 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:36:17.860 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:17.860 7f6e771f4a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:36:17.860 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:17.860 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:36:17.860 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:36:17.860 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:17.860 7f6e771f4a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:36:17.861 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:17.861 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:17.861 7f6e771f4a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:36:17.861 7f6e771f4a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:21.864 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:36:25.180 7f6e7369a6d8 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:36:26.869 7f6e71802978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:36:26.870 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:36:26.870 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:26.870 7f6e71802978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:36:26.870 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:26.870 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:36:26.870 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:36:26.870 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:26.870 7f6e71802978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:36:26.870 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:26.871 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:26.871 7f6e71802978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:36:26.871 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:30.874 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:36:35.878 7f6e71802978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:36:35.878 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:36:35.878 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:35.878 7f6e71802978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:36:35.878 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:35.878 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:36:35.879 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:36:35.879 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:35.879 7f6e71802978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:36:35.879 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:35.879 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:35.879 7f6e71802978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:36:35.879 7f6e71802978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:39.883 7f6e736a62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:36:40.517 7f6e771ee780 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:36:44.889 7f6e736a62e8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:36:44.889 7f6e736a62e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:36:44.889 7f6e736a62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:44.889 7f6e736a62e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:36:44.889 7f6e736a62e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:44.889 7f6e736a62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:36:44.890 7f6e736a62e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:36:44.890 7f6e736a62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:44.890 7f6e736a62e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:36:44.890 7f6e736a62e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:44.890 7f6e736a62e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:44.890 7f6e736a62e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:36:44.890 7f6e736a62e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:48.890 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:36:53.896 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:36:53.897 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:36:53.897 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:53.897 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:36:53.897 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:53.897 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:36:53.898 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:36:53.898 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:53.899 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:36:53.899 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:53.899 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:53.899 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:36:53.899 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:57.900 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:37:02.902 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:37:02.902 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:37:02.903 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:02.903 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:37:02.903 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:02.903 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:37:02.904 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:37:02.904 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:02.904 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:37:02.904 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:02.904 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:02.904 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:37:02.904 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:03.558 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51449920, partition_space_available_1k=29640564, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29496, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94660, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59844, cpu_share=0.01, mem_res=39652, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15367712, free=9333020, buffers=997988, cached=6966400, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.23, five_min_avg=3.52, fifteen_min_avg=3.63, cpu_share=5.38, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:37:03.558 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:03.558 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51449920 partition_space_available_1k = 29640564 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29496 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94660 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59844 cpu_share = 0.01 mem_res = 39652 >> } system_mem_usage = << total = 32665120 used = 15367712 free = 9333020 buffers = 997988 cached = 6966400 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 5.23 five_min_avg = 3.52 fifteen_min_avg = 3.63 cpu_share = 5.38 node_type = analytics-node >> >> 06/02/2025 02:37:03.558 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:06.905 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:37:11.911 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:37:11.911 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:37:11.911 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:11.912 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:37:11.912 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:11.912 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:37:11.912 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:37:11.912 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:11.912 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:37:11.912 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:11.912 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:11.913 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:37:11.913 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:15.914 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:37:20.918 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:37:20.918 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:37:20.919 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:20.919 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:37:20.919 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:20.919 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:37:20.919 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:37:20.919 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:20.919 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:37:20.919 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:20.919 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:20.919 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:37:20.920 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:24.922 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:37:29.929 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:37:29.929 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:37:29.929 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:29.929 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:37:29.929 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:29.929 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:37:29.930 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:37:29.930 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:29.930 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:37:29.931 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:29.931 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:29.931 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:37:29.931 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:33.930 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:37:38.937 7f6e72dde240 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:37:38.937 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:37:38.937 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:38.937 7f6e72dde240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:37:38.937 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:38.937 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:37:38.938 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:37:38.938 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:38.938 7f6e72dde240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:37:38.938 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:38.938 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:38.938 7f6e72dde240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:37:38.938 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:42.940 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:37:47.947 7f6e72dde240 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:37:47.947 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:37:47.947 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:47.947 7f6e72dde240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:37:47.947 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:47.948 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:37:47.948 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:37:47.948 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:47.948 7f6e72dde240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:37:47.948 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:47.948 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:47.948 7f6e72dde240 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:37:47.948 7f6e72dde240 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:51.951 7f6e736a7e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:37:56.957 7f6e736a7e10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:37:56.958 7f6e736a7e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:37:56.958 7f6e736a7e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:56.958 7f6e736a7e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:37:56.958 7f6e736a7e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:56.958 7f6e736a7e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:37:56.958 7f6e736a7e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:37:56.958 7f6e736a7e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:56.959 7f6e736a7e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:37:56.959 7f6e736a7e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:56.959 7f6e736a7e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:56.959 7f6e736a7e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:37:56.959 7f6e736a7e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:00.959 7f6e72dc3b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:38:03.557 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51452476, partition_space_available_1k=29638008, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29496, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94708, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59900, cpu_share=0.01, mem_res=39652, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15365512, free=9332296, buffers=997988, cached=6969324, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.81, five_min_avg=3.44, fifteen_min_avg=3.6, cpu_share=2.5, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:38:05.964 7f6e72dc3b70 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:38:05.965 7f6e72dc3b70 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:38:05.965 7f6e72dc3b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:05.965 7f6e72dc3b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:38:05.965 7f6e72dc3b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:05.965 7f6e72dc3b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:05.965 7f6e72dc3b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51452476 partition_space_available_1k = 29638008 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29496 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94708 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59900 cpu_share = 0.01 mem_res = 39652 >> } system_mem_usage = << total = 32665120 used = 15365512 free = 9332296 buffers = 997988 cached = 6969324 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 3.81 five_min_avg = 3.44 fifteen_min_avg = 3.6 cpu_share = 2.5 node_type = analytics-node >> >> 06/02/2025 02:38:05.965 7f6e72dc3b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:05.965 7f6e72dc3b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:38:05.966 7f6e72dc3b70 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:38:05.966 7f6e72dc3b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:05.966 7f6e72dc3b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:38:05.966 7f6e72dc3b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:05.966 7f6e72dc3b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:05.966 7f6e72dc3b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:38:05.966 7f6e72dc3b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:09.968 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:38:14.975 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:38:14.975 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:38:14.975 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:14.975 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:38:14.975 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:14.975 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:38:14.976 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:38:14.976 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:14.976 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:38:14.976 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:14.976 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:14.976 7f6e756a5a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:38:14.976 7f6e756a5a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:18.975 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:38:23.981 7f6e7369a2e8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:38:23.982 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:38:23.982 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:23.982 7f6e7369a2e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:38:23.982 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:23.982 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:38:23.983 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:38:23.983 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:23.983 7f6e7369a2e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:38:23.983 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:23.983 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:23.983 7f6e7369a2e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:38:23.983 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:24.613 7f6e7369a2e8 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:38:27.985 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:38:32.990 7f6e7369a2e8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:38:32.990 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:38:32.990 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:32.990 7f6e7369a2e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:38:32.990 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:32.990 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:38:32.991 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:38:32.991 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:32.991 7f6e7369a2e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:38:32.991 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:32.991 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:32.991 7f6e7369a2e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:38:32.991 7f6e7369a2e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:36.993 7f6e771eaf60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:38:41.999 7f6e771eaf60 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:38:41.999 7f6e771eaf60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:38:41.999 7f6e771eaf60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:41.999 7f6e771eaf60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:38:41.999 7f6e771eaf60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:41.999 7f6e771eaf60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:38:42.000 7f6e771eaf60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:38:42.000 7f6e771eaf60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:42.000 7f6e771eaf60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:38:42.000 7f6e771eaf60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:42.000 7f6e771eaf60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:42.000 7f6e771eaf60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:38:42.000 7f6e771eaf60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:46.001 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:38:51.005 7f6e76eb9f60 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:38:51.005 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:38:51.005 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:51.005 7f6e76eb9f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:38:51.005 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:51.005 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:38:51.006 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:38:51.006 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:51.006 7f6e76eb9f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:38:51.006 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:51.007 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:51.007 7f6e76eb9f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:38:51.008 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:55.009 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:39:00.012 7f6e76eb9f60 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:39:00.012 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:39:00.013 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:00.013 7f6e76eb9f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:39:00.013 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:00.013 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:39:00.014 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:39:00.014 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:00.014 7f6e76eb9f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:39:00.014 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:00.014 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:00.014 7f6e76eb9f60 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:39:00.014 7f6e76eb9f60 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:03.559 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51458492, partition_space_available_1k=29631992, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29500, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94756, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59888, cpu_share=0.01, mem_res=39652, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15351864, free=9343808, buffers=997996, cached=6971452, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.96, five_min_avg=3.27, fifteen_min_avg=3.53, cpu_share=2.62, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:39:03.559 7f6e77200eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:03.560 7f6e77200eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51458492 partition_space_available_1k = 29631992 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29500 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94756 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59888 cpu_share = 0.01 mem_res = 39652 >> } system_mem_usage = << total = 32665120 used = 15351864 free = 9343808 buffers = 997996 cached = 6971452 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 2.96 five_min_avg = 3.27 fifteen_min_avg = 3.53 cpu_share = 2.62 node_type = analytics-node >> >> 06/02/2025 02:39:03.560 7f6e77200eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:04.013 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:39:09.019 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:39:09.019 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:39:09.019 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:09.020 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:39:09.020 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:09.020 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:39:09.020 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:39:09.021 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:09.021 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:39:09.021 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:09.021 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:09.021 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:39:09.021 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:13.023 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:39:18.029 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:39:18.029 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:39:18.029 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:18.029 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:39:18.029 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:18.029 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:39:18.030 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:39:18.031 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:18.031 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:39:18.031 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:18.031 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:18.031 7f6e73a77eb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:39:18.031 7f6e73a77eb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:22.031 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:39:27.037 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:39:27.037 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:39:27.037 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:27.037 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:39:27.037 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:27.037 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:39:27.038 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:39:27.038 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:27.038 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:39:27.038 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:27.038 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:27.038 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:39:27.038 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:31.041 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:39:36.046 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:39:36.046 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:39:36.046 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:36.046 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:39:36.046 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:36.046 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:39:36.047 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:39:36.047 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:36.047 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:39:36.047 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:36.047 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:36.047 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:39:36.047 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:40.049 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:39:45.056 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:39:45.056 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:39:45.056 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:45.056 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:39:45.056 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:45.057 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:39:45.057 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:39:45.057 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:45.057 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:39:45.057 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:45.057 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:45.058 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:39:45.058 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:49.059 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:39:54.064 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:39:54.064 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:39:54.064 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:54.064 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:39:54.064 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:54.064 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:39:54.065 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:39:54.065 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:54.065 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:39:54.065 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:54.065 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:54.065 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:39:54.065 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:58.067 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:40:03.071 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:40:03.071 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:40:03.071 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:03.071 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:40:03.072 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:03.072 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:40:03.073 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:40:03.073 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:03.073 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:40:03.073 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:03.073 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:03.074 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:40:03.074 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:03.563 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51459032, partition_space_available_1k=29631452, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29500, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94756, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59916, cpu_share=0.01, mem_res=39652, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15382856, free=9303864, buffers=998000, cached=6980400, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.25, five_min_avg=3.01, fifteen_min_avg=3.42, cpu_share=2.62, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:40:03.563 7f6e7367e048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:03.563 7f6e7367e048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51459032 partition_space_available_1k = 29631452 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29500 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94756 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59916 cpu_share = 0.01 mem_res = 39652 >> } system_mem_usage = << total = 32665120 used = 15382856 free = 9303864 buffers = 998000 cached = 6980400 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 2.25 five_min_avg = 3.01 fifteen_min_avg = 3.42 cpu_share = 2.62 node_type = analytics-node >> >> 06/02/2025 02:40:03.563 7f6e7367e048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:07.072 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:40:12.079 7f6e72ddea20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:40:12.079 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:40:12.079 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:12.079 7f6e72ddea20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:40:12.079 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:12.079 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:40:12.080 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:40:12.080 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:12.080 7f6e72ddea20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:40:12.080 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:12.080 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:12.080 7f6e72ddea20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:40:12.080 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:16.081 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:40:17.886 7f6e72e25e10 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:40:21.083 7f6e72ddea20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:40:21.083 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:40:21.083 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:21.083 7f6e72ddea20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:40:21.083 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:21.083 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:40:21.084 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:40:21.084 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:21.084 7f6e72ddea20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:40:21.084 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:21.084 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:21.084 7f6e72ddea20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:40:21.084 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:25.085 7f6e718028d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:40:28.320 7f6e72ddea20 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:40:30.088 7f6e718028d0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:40:30.088 7f6e718028d0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:40:30.088 7f6e718028d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:30.088 7f6e718028d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:40:30.088 7f6e718028d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:30.088 7f6e718028d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:40:30.089 7f6e718028d0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:40:30.089 7f6e718028d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:30.089 7f6e718028d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:40:30.089 7f6e718028d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:30.089 7f6e718028d0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:30.089 7f6e718028d0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:40:30.089 7f6e718028d0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:34.092 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:40:39.097 7f6e72ddea20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:40:39.097 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:40:39.097 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:39.097 7f6e72ddea20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:40:39.098 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:39.098 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:40:39.098 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:40:39.098 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:39.098 7f6e72ddea20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:40:39.098 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:39.098 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:39.098 7f6e72ddea20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:40:39.098 7f6e72ddea20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:40.555 7f6e766dbc18 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:40:43.100 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:40:48.105 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:40:48.105 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:40:48.105 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:48.105 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:40:48.105 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:48.105 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:40:48.106 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:40:48.106 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:48.106 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:40:48.106 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:48.106 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:48.106 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:40:48.106 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:52.107 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:40:57.113 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:40:57.113 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:40:57.113 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:57.113 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:40:57.113 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:57.113 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:40:57.114 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:40:57.114 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:57.114 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:40:57.114 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:57.114 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:57.114 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:40:57.114 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:01.115 7f6e77208780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:41:03.575 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51469720, partition_space_available_1k=29620764, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29508, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94816, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59968, cpu_share=0.01, mem_res=39652, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15446832, free=9218884, buffers=998008, cached=7001396, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=7.69, five_min_avg=4.43, fifteen_min_avg=3.88, cpu_share=5.38, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:41:06.118 7f6e77208780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:41:06.118 7f6e77208780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:41:06.118 7f6e77208780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:06.118 7f6e77208780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:41:06.118 7f6e77208780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:06.119 7f6e77208780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:06.119 7f6e77208780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51469720 partition_space_available_1k = 29620764 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29508 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94816 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59968 cpu_share = 0.01 mem_res = 39652 >> } system_mem_usage = << total = 32665120 used = 15446832 free = 9218884 buffers = 998008 cached = 7001396 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 7.69 five_min_avg = 4.43 fifteen_min_avg = 3.88 cpu_share = 5.38 node_type = analytics-node >> >> 06/02/2025 02:41:06.119 7f6e77208780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:06.119 7f6e77208780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:41:06.119 7f6e77208780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:41:06.119 7f6e77208780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:06.120 7f6e77208780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:41:06.120 7f6e77208780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:06.120 7f6e77208780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:06.120 7f6e77208780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:41:06.120 7f6e77208780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:10.122 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:41:15.128 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:41:15.129 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:41:15.129 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:15.129 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:41:15.129 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:15.129 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:41:15.129 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:41:15.130 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:15.130 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:41:15.130 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:15.130 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:15.130 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:41:15.130 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:19.132 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:41:24.137 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:41:24.138 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:41:24.138 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:24.138 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:41:24.138 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:24.138 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:41:24.138 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:41:24.139 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:24.139 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:41:24.139 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:24.139 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:24.140 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:41:24.140 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:28.142 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:41:33.149 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:41:33.149 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:41:33.150 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:33.150 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:41:33.150 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:33.150 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:41:33.151 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:41:33.151 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:33.151 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:41:33.151 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:33.151 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:33.151 7f6e7bfa7a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:41:33.152 7f6e7bfa7a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:37.149 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:41:42.156 7f6e72dba048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:41:42.156 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:41:42.156 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:42.156 7f6e72dba048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:41:42.156 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:42.157 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:41:42.157 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:41:42.157 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:42.157 7f6e72dba048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:41:42.157 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:42.157 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:42.157 7f6e72dba048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:41:42.157 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:46.161 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:41:51.163 7f6e72dba048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:41:51.163 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:41:51.163 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:51.163 7f6e72dba048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:41:51.163 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:51.164 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:41:51.164 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:41:51.164 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:51.164 7f6e72dba048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:41:51.164 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:51.164 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:51.164 7f6e72dba048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:41:51.164 7f6e72dba048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:55.167 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:42:00.169 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:42:00.169 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:42:00.169 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:00.170 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:42:00.170 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:00.170 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:42:00.171 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:42:00.171 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:00.171 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:42:00.171 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:00.171 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:00.171 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:42:00.171 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:03.564 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51471600, partition_space_available_1k=29618884, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29504, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94844, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59960, cpu_share=0.01, mem_res=39652, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15432964, free=9229872, buffers=998008, cached=7004276, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.02, five_min_avg=3.98, fifteen_min_avg=3.76, cpu_share=2.62, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:42:03.565 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:03.565 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51471600 partition_space_available_1k = 29618884 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29504 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94844 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59960 cpu_share = 0.01 mem_res = 39652 >> } system_mem_usage = << total = 32665120 used = 15432964 free = 9229872 buffers = 998008 cached = 7004276 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 4.02 five_min_avg = 3.98 fifteen_min_avg = 3.76 cpu_share = 2.62 node_type = analytics-node >> >> 06/02/2025 02:42:03.565 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:04.170 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:42:09.174 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:42:09.175 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:42:09.175 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:09.175 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:42:09.175 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:09.175 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:42:09.175 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:42:09.176 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:09.176 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:42:09.176 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:09.176 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:09.176 7f6e7368beb8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:42:09.176 7f6e7368beb8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:13.178 7f6e718024e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:42:18.184 7f6e718024e0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:42:18.184 7f6e718024e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:42:18.184 7f6e718024e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:18.184 7f6e718024e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:42:18.184 7f6e718024e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:18.184 7f6e718024e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:42:18.185 7f6e718024e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:42:18.185 7f6e718024e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:18.185 7f6e718024e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:42:18.185 7f6e718024e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:18.185 7f6e718024e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:18.185 7f6e718024e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:42:18.185 7f6e718024e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:22.185 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:42:27.191 7f6e72dc2048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:42:27.191 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:42:27.192 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:27.192 7f6e72dc2048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:42:27.192 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:27.192 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:42:27.192 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:42:27.192 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:27.192 7f6e72dc2048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:42:27.192 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:27.193 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:27.193 7f6e72dc2048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:42:27.193 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:31.194 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:42:36.200 7f6e72dc2048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:42:36.200 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:42:36.201 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:36.201 7f6e72dc2048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:42:36.201 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:36.201 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:42:36.201 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:42:36.201 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:36.201 7f6e72dc2048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:42:36.201 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:36.201 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:36.202 7f6e72dc2048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:42:36.202 7f6e72dc2048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:40.204 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:42:45.210 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:42:45.210 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:42:45.210 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:45.210 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:42:45.210 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:45.211 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:42:45.211 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:42:45.211 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:45.211 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:42:45.211 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:45.211 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:45.211 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:42:45.211 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:49.214 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:42:54.219 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:42:54.219 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:42:54.219 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:54.219 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:42:54.220 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:54.220 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:42:54.220 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:42:54.220 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:54.220 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:42:54.220 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:54.220 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:54.220 7f6e736942e8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:42:54.220 7f6e736942e8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:58.224 7f6e7366b048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:43:03.228 7f6e7366b048 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:43:03.228 7f6e7366b048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:43:03.228 7f6e7366b048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:03.228 7f6e7366b048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:43:03.228 7f6e7366b048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:03.229 7f6e7366b048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:43:03.229 7f6e7366b048 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:43:03.229 7f6e7366b048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:03.229 7f6e7366b048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:43:03.229 7f6e7366b048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:03.230 7f6e7366b048 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:03.230 7f6e7366b048 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:43:03.230 7f6e7366b048 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:03.566 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51472964, partition_space_available_1k=29617520, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29508, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94844, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=59984, cpu_share=0.01, mem_res=39652, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15416452, free=9243876, buffers=998008, cached=7006784, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.41, five_min_avg=3.83, fifteen_min_avg=3.72, cpu_share=2.62, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:43:03.566 7f6e736ae780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:03.566 7f6e736ae780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51472964 partition_space_available_1k = 29617520 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29508 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94844 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 59984 cpu_share = 0.01 mem_res = 39652 >> } system_mem_usage = << total = 32665120 used = 15416452 free = 9243876 buffers = 998008 cached = 7006784 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 3.41 five_min_avg = 3.83 fifteen_min_avg = 3.72 cpu_share = 2.62 node_type = analytics-node >> >> 06/02/2025 02:43:03.566 7f6e736ae780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:07.229 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:43:12.235 7f6e72dc2b70 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:43:12.235 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:43:12.235 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:12.235 7f6e72dc2b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:43:12.235 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:12.235 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:43:12.236 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:43:12.236 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:12.236 7f6e72dc2b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:43:12.236 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:12.236 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:12.236 7f6e72dc2b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:43:12.236 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:16.240 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:43:16.609 7f6e71802978 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:43:21.245 7f6e72dc2b70 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:43:21.245 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:43:21.245 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:21.245 7f6e72dc2b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:43:21.245 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:21.245 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:43:21.245 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:43:21.246 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:21.246 7f6e72dc2b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:43:21.246 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:21.246 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:21.246 7f6e72dc2b70 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:43:21.246 7f6e72dc2b70 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:25.248 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:43:30.254 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:43:30.255 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:43:30.255 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:30.255 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:43:30.255 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:30.255 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:43:30.256 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:43:30.256 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:30.257 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:43:30.258 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:30.258 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:30.258 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:43:30.259 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:34.257 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:43:39.261 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:43:39.261 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:43:39.261 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:39.261 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:43:39.261 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:39.261 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:43:39.262 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:43:39.262 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:39.262 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:43:39.262 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:39.262 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:39.262 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:43:39.262 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:43.264 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:43:48.269 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:43:48.269 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:43:48.269 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:48.269 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:43:48.269 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:48.269 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:43:48.270 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:43:48.270 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:48.270 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:43:48.270 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:48.270 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:48.271 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:43:48.271 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:52.269 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:43:57.275 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:43:57.275 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:43:57.275 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:57.275 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:43:57.275 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:57.275 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:43:57.276 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:43:57.276 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:57.276 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:43:57.276 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:57.276 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:57.276 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:43:57.276 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:01.277 7f6e72ddec18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:44:03.567 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51479172, partition_space_available_1k=29611312, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29508, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94904, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=60040, cpu_share=0.01, mem_res=39652, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15402296, free=9250896, buffers=998008, cached=7013920, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.72, five_min_avg=3.59, fifteen_min_avg=3.65, cpu_share=2.75, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:44:06.282 7f6e72ddec18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:44:06.282 7f6e72ddec18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:44:06.282 7f6e72ddec18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:06.282 7f6e72ddec18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:44:06.283 7f6e72ddec18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:06.283 7f6e72ddec18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:06.283 7f6e72ddec18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51479172 partition_space_available_1k = 29611312 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29508 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94904 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 60040 cpu_share = 0.01 mem_res = 39652 >> } system_mem_usage = << total = 32665120 used = 15402296 free = 9250896 buffers = 998008 cached = 7013920 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 2.72 five_min_avg = 3.59 fifteen_min_avg = 3.65 cpu_share = 2.75 node_type = analytics-node >> >> 06/02/2025 02:44:06.283 7f6e72ddec18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:06.283 7f6e72ddec18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:44:06.283 7f6e72ddec18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:44:06.283 7f6e72ddec18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:06.283 7f6e72ddec18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:44:06.285 7f6e72ddec18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:06.285 7f6e72ddec18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:06.285 7f6e72ddec18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:44:06.285 7f6e72ddec18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:10.285 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:44:15.291 7f6e736a6a20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:44:15.292 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:44:15.292 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:15.292 7f6e736a6a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:44:15.292 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:15.292 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:44:15.293 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:44:15.293 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:15.293 7f6e736a6a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:44:15.293 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:15.294 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:15.294 7f6e736a6a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:44:15.294 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:19.296 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:44:24.300 7f6e736a6a20 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:44:24.300 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:44:24.301 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:24.301 7f6e736a6a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:44:24.301 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:24.301 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:44:24.301 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:44:24.301 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:24.302 7f6e736a6a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:44:24.302 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:24.302 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:24.302 7f6e736a6a20 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:44:24.302 7f6e736a6a20 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:28.302 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:44:33.309 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:44:33.309 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:44:33.309 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:33.309 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:44:33.309 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:33.309 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:44:33.310 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:44:33.310 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:33.310 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:44:33.310 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:33.310 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:33.310 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:44:33.310 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:37.312 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:44:42.319 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:44:42.319 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:44:42.319 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:42.319 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:44:42.319 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:42.320 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:44:42.320 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:44:42.320 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:42.320 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:44:42.320 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:42.321 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:42.321 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:44:42.321 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:46.323 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:44:51.329 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:44:51.330 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:44:51.330 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:51.330 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:44:51.330 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:51.330 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:44:51.330 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:44:51.331 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:51.331 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:44:51.331 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:51.331 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:51.331 7f6e7cb500f0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:44:51.331 7f6e7cb500f0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:55.333 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:00.336 7f6e771f4ac8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:45:00.336 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:45:00.336 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:00.337 7f6e771f4ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:45:00.337 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:00.337 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:45:00.337 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:45:00.338 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:00.338 7f6e771f4ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:45:00.338 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:00.338 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:00.338 7f6e771f4ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:45:00.338 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:03.571 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51479808, partition_space_available_1k=29610676, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29504, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94932, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=60028, cpu_share=0.01, mem_res=39652, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15432888, free=9218756, buffers=998008, cached=7015468, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.58, five_min_avg=3.38, fifteen_min_avg=3.57, cpu_share=2.75, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:45:03.571 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:03.572 7f6e771f4ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51479808 partition_space_available_1k = 29610676 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29504 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94932 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 60028 cpu_share = 0.01 mem_res = 39652 >> } system_mem_usage = << total = 32665120 used = 15432888 free = 9218756 buffers = 998008 cached = 7015468 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 2.58 five_min_avg = 3.38 fifteen_min_avg = 3.57 cpu_share = 2.75 node_type = analytics-node >> >> 06/02/2025 02:45:03.572 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:04.337 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:09.339 7f6e771f4ac8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:45:09.340 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:45:09.340 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:09.340 7f6e771f4ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:45:09.340 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:09.340 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:45:09.341 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:45:09.342 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:09.342 7f6e771f4ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:45:09.342 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:09.342 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:09.342 7f6e771f4ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:45:09.342 7f6e771f4ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:13.342 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:18.349 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:45:18.349 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:45:18.349 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:18.349 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:45:18.349 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:18.349 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:45:18.350 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:45:18.350 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:18.350 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:45:18.350 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:18.350 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:18.350 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:45:18.350 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:22.354 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:27.359 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:45:27.360 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:45:27.360 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:27.360 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:45:27.360 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:27.360 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:45:27.361 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:45:27.361 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:27.361 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:45:27.361 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:27.361 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:27.362 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:45:27.362 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:31.364 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:36.370 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:45:36.370 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:45:36.370 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:36.370 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:45:36.371 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:36.371 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:45:36.371 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:45:36.371 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:36.371 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:45:36.371 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:36.372 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:36.372 7f6e766dbc18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:45:36.372 7f6e766dbc18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:40.371 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:45.378 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:45:45.379 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:45:45.379 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:45.379 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:45:45.379 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:45.379 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:45:45.380 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:45:45.380 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:45.380 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:45:45.380 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:45.380 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:45.380 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:45:45.380 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:49.380 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:54.382 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:45:54.383 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:45:54.383 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:54.383 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:45:54.383 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:54.383 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:45:54.383 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:45:54.384 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:54.384 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:45:54.384 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:54.384 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:54.384 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:45:54.384 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:55.722 7f6e75787588 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:45:58.385 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:46:03.389 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:46:03.389 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:46:03.389 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:03.390 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:46:03.390 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:03.390 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:46:03.390 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:46:03.390 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:03.390 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:46:03.391 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:03.391 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:03.391 7f6e75787588 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:46:03.391 7f6e75787588 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:03.580 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51488608, partition_space_available_1k=29601876, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29508, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94936, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=60052, cpu_share=0.01, mem_res=39652, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15515656, free=9113336, buffers=998008, cached=7038120, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.54, five_min_avg=4.53, fifteen_min_avg=3.96, cpu_share=5.38, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:46:03.581 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:03.581 7f6e72dba4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51488608 partition_space_available_1k = 29601876 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29508 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94936 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 60052 cpu_share = 0.01 mem_res = 39652 >> } system_mem_usage = << total = 32665120 used = 15515656 free = 9113336 buffers = 998008 cached = 7038120 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 6.54 five_min_avg = 4.53 fifteen_min_avg = 3.96 cpu_share = 5.38 node_type = analytics-node >> >> 06/02/2025 02:46:03.581 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:05.794 7f6e72dba4e0 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:46:07.391 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:46:12.397 7f6e72dba4e0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:46:12.397 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:46:12.397 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:12.397 7f6e72dba4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:46:12.397 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:12.397 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:46:12.398 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:46:12.398 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:12.398 7f6e72dba4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:46:12.398 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:12.398 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:12.399 7f6e72dba4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:46:12.399 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:16.401 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:46:16.733 7f6e766dbc18 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:46:21.406 7f6e72dba4e0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:46:21.407 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:46:21.407 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:21.407 7f6e72dba4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:46:21.407 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:21.407 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:46:21.408 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:46:21.408 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:21.408 7f6e72dba4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:46:21.408 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:21.409 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:21.409 7f6e72dba4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:46:21.409 7f6e72dba4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:25.408 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:46:30.413 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:46:30.413 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:46:30.413 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:30.413 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:46:30.413 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:30.413 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:46:30.414 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:46:30.414 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:30.414 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:46:30.414 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:30.414 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:30.414 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:46:30.414 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:34.417 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:46:39.419 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:46:39.419 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:46:39.419 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:39.419 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:46:39.419 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:39.419 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:46:39.420 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:46:39.420 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:39.420 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:46:39.420 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:39.420 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:39.420 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:46:39.420 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:43.424 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:46:48.430 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:46:48.430 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:46:48.430 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:48.430 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:46:48.430 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:48.431 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:46:48.431 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:46:48.431 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:48.431 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:46:48.431 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:48.431 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:48.431 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:46:48.431 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:52.434 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:46:57.438 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:46:57.439 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:46:57.439 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:57.439 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:46:57.439 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:57.439 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:46:57.440 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:46:57.440 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:57.440 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:46:57.440 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:57.441 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:57.441 7f6e7af0e780 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:46:57.441 7f6e7af0e780 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:01.441 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:47:03.573 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51492148, partition_space_available_1k=29598336, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29504, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=94988, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=60108, cpu_share=0.01, mem_res=39652, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15442196, free=9185564, buffers=998012, cached=7039348, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.88, five_min_avg=4.12, fifteen_min_avg=3.86, cpu_share=3.12, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:47:06.446 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:47:06.446 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:47:06.447 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:06.447 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:47:06.447 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:06.447 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:06.447 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51492148 partition_space_available_1k = 29598336 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29504 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 94988 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 60108 cpu_share = 0.01 mem_res = 39652 >> } system_mem_usage = << total = 32665120 used = 15442196 free = 9185564 buffers = 998012 cached = 7039348 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 3.88 five_min_avg = 4.12 fifteen_min_avg = 3.86 cpu_share = 3.12 node_type = analytics-node >> >> 06/02/2025 02:47:06.448 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:06.448 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:47:06.448 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:47:06.448 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:06.449 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:47:06.449 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:06.449 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:06.449 7f6e75d14e10 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:47:06.449 7f6e75d14e10 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:10.446 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:47:15.450 7f6e72dde4e0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:47:15.451 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:47:15.451 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:15.451 7f6e72dde4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:47:15.451 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:15.451 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:47:15.451 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:47:15.451 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:15.451 7f6e72dde4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:47:15.451 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:15.452 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:15.452 7f6e72dde4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:47:15.452 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:19.453 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:47:24.455 7f6e72dde4e0 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:47:24.455 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:47:24.455 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:24.456 7f6e72dde4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:47:24.456 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:24.456 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:47:24.456 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:47:24.456 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:24.457 7f6e72dde4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:47:24.457 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:24.457 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:24.457 7f6e72dde4e0 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:47:24.457 7f6e72dde4e0 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:27.189 7f6e72dc3828 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map 06/02/2025 02:47:28.457 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:47:33.463 7f6e72dc3828 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:47:33.463 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:47:33.463 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:33.463 7f6e72dc3828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:47:33.463 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:33.463 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:47:33.464 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:47:33.464 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:33.464 7f6e72dc3828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:47:33.464 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:33.464 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:33.464 7f6e72dc3828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:47:33.465 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:37.467 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:47:42.471 7f6e72dc3828 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:47:42.472 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:47:42.472 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:42.472 7f6e72dc3828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:47:42.472 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:42.472 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:47:42.472 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:47:42.472 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:42.473 7f6e72dc3828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:47:42.473 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:42.473 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:42.473 7f6e72dc3828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:47:42.473 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:46.475 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:47:51.481 7f6e72dc3828 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:47:51.481 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:47:51.481 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:51.481 7f6e72dc3828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:47:51.481 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:51.481 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:47:51.482 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:47:51.482 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:51.482 7f6e72dc3828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:47:51.482 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:51.482 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:51.482 7f6e72dc3828 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:47:51.482 7f6e72dc3828 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:55.481 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:48:00.484 7f6e72245c18 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:48:00.484 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:48:00.485 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:00.485 7f6e72245c18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:48:00.485 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:00.485 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:48:00.485 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:48:00.486 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:00.486 7f6e72245c18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:48:00.486 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:00.486 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:00.486 7f6e72245c18 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:48:00.486 7f6e72245c18 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:03.578 7f6e75d6f308 [contrail-analytics-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51493924, partition_space_available_1k=29596560, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-collector': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=29512, cpu_share=0.0, mem_res=13592, _EventManager__key='contrail-collector'), 'contrail-analytics-api': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=95032, cpu_share=0.0, mem_res=84752, _EventManager__key='contrail-analytics-api'), 'contrail-analytics-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=60104, cpu_share=0.01, mem_res=39652, _EventManager__key='contrail-analytics-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15429392, free=9195076, buffers=998016, cached=7042636, node_type='analytics-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.45, five_min_avg=3.98, fifteen_min_avg=3.83, cpu_share=3.0, node_type='analytics-node'), running_package_version=None, installed_package_version=None, _table='ObjectCollectorInfo') 06/02/2025 02:48:03.579 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:03.579 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51493924 partition_space_available_1k = 29596560 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { contrail-collector : << mem_virt = 29512 cpu_share = 0.0 mem_res = 13592 >>contrail-analytics-api : << mem_virt = 95032 cpu_share = 0.0 mem_res = 84752 >>contrail-analytics-nodemgr : << mem_virt = 60104 cpu_share = 0.01 mem_res = 39652 >> } system_mem_usage = << total = 32665120 used = 15429392 free = 9195076 buffers = 998016 cached = 7042636 node_type = analytics-node >> system_cpu_usage = << one_min_avg = 3.45 five_min_avg = 3.98 fifteen_min_avg = 3.83 cpu_share = 3.0 node_type = analytics-node >> >> 06/02/2025 02:48:03.579 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:04.485 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:48:09.490 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:48:09.490 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:48:09.490 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:09.490 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:48:09.491 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:09.491 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:48:09.491 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:48:09.491 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:09.491 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:48:09.491 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:09.491 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:09.492 7f6e756aa978 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:48:09.492 7f6e756aa978 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:13.492 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:48:18.499 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:48:18.499 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:48:18.499 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:18.499 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:48:18.499 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:18.502 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:48:18.503 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:48:18.503 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:18.503 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:48:18.503 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:18.503 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:18.503 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:48:18.503 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:22.503 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:48:27.508 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:48:27.508 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:48:27.508 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:27.508 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:48:27.508 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:27.508 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:48:27.509 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:48:27.509 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:27.509 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-analytics-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:48:27.509 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:27.509 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:27.509 7f6e766d1ac8 [contrail-analytics-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Analytics:contrail-analytics-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8104 start_time = 1748829958142916 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:48:27.509 7f6e766d1ac8 [contrail-analytics-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:29.018 7f6e766d1ac8 [contrail-analytics-nodemgr] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:NodeStatusUVE] map