06/02/2025 02:06:43.849 7f57813cbe48 [contrail-control-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 06/02/2025 02:06:43.849 7f57813cbe48 [contrail-control-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 06/02/2025 02:06:43.849 7f57813cbe48 [contrail-control-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 06/02/2025 02:06:43.862 7f577e2e9048 [contrail-control-nodemgr] [ERROR]: Cannot write http_port 8101 to /tmp/contrail-control-nodemgr.0.http_port 06/02/2025 02:06:43.863 7f577e2e9048 [contrail-control-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8101 06/02/2025 02:06:43.864 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Processing event[EvStart] in state[none] 06/02/2025 02:06:43.869 7f577e2f0978 [contrail-control-nodemgr] [DEBUG]: Add uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:06:43.869 7f577e2f0978 [contrail-control-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/02/2025 02:06:43.869 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 06/02/2025 02:06:43.870 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:43.870 7f577e2f0978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/02/2025 02:06:43.870 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:43.870 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:43.870 7f577e2f0978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 06/02/2025 02:06:43.870 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:44.274 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'contrail-control', 'group': 'contrail-control', 'pid': 93092353574206810711741429414100794238006717385525638630125476230589025082443, 'start': '1748830000000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-dns', 'group': 'contrail-dns', 'pid': 55679479010560454021876438189723212518592630044050581977503551425462817110063, 'start': '1748830000000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-named', 'group': 'contrail-named', 'pid': 80501009525279241953224844737804863183837413272099316519129426927856937837138, 'start': '1748830000000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-control-nodemgr', 'group': 'contrail-control-nodemgr', 'pid': 25997495725803767495257781088728949071674547513459772288172611309353605491195, 'start': '1748830000000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 06/02/2025 02:06:44.474 7f57813cbe48 [contrail-control-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-control-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-542-1', _node_type='Control', _instance_id='0', _timestamp=1748830004474387, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-control-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter'), _seqnum=0) 06/02/2025 02:06:44.475 7f57813cbe48 [contrail-control-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-control-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-542-1', _node_type='Control', _instance_id='0', _timestamp=1748830004475057, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='contrail-control', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1748830000000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-dns', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1748830000000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-named', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1748830000000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-control-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1748830000000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-02 00:52:12.596288", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9311-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter'), _seqnum=0) 06/02/2025 02:06:44.478 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:44.479 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-02 00:52:12.596288", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9311-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 06/02/2025 02:06:44.479 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:44.479 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:44.479 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/02/2025 02:06:44.479 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:44.479 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:44.482 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 deleted = False process_info = [ << process_name = contrail-control process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1748830000000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-dns process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1748830000000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-named process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1748830000000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-control-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1748830000000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-02 00:52:12.596288", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9311-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 06/02/2025 02:06:44.483 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:46.910 7f57811c84e0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:06:47.870 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:06:52.498 7f57813cbe48 [contrail-control-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-control-nodemgr', _source='cn-jenkins-deploy-platform-juju-os-542-1', _node_type='Control', _instance_id='0', _timestamp=1748830012497977, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='contrail-control', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1748830000000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-dns', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1748830000000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-named', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1748830000000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-control-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1748830000000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-02 00:52:12.596288", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9311-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter'), _seqnum=0) 06/02/2025 02:06:52.499 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51186452, partition_space_available_1k=29904032, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17860, cpu_share=0.0, mem_res=4304, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=19856, cpu_share=0.0, mem_res=3764, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=37012, cpu_share=0.0, mem_res=32096, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41820, cpu_share=0.0, mem_res=39628, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=13946444, free=12807436, buffers=933680, cached=4977560, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=7.92, five_min_avg=4.26, fifteen_min_avg=2.79, cpu_share=0, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:06:52.873 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:06:52.898 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:06:52.898 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:52.899 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:06:52.899 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:52.899 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:52.899 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 deleted = False process_info = [ << process_name = contrail-control process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1748830000000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-dns process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1748830000000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-named process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1748830000000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-control-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1748830000000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-02 00:52:12.596288", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9311-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 06/02/2025 02:06:52.899 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:52.899 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:52.899 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51186452 partition_space_available_1k = 29904032 percentage_partition_space_used = 63 >> } all_core_file_list = [ ] process_mem_cpu_usage = { contrail-control : << mem_virt = 17860 cpu_share = 0.0 mem_res = 4304 >>contrail-dns : << mem_virt = 19856 cpu_share = 0.0 mem_res = 3764 >>contrail-named : << mem_virt = 37012 cpu_share = 0.0 mem_res = 32096 >>contrail-control-nodemgr : << mem_virt = 41820 cpu_share = 0.0 mem_res = 39628 >> } system_mem_usage = << total = 32665120 used = 13946444 free = 12807436 buffers = 933680 cached = 4977560 node_type = control-node >> system_cpu_usage = << one_min_avg = 7.92 five_min_avg = 4.26 fifteen_min_avg = 2.79 cpu_share = 0 node_type = control-node >> >> 06/02/2025 02:06:52.899 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:06:52.899 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:06:52.900 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:06:52.900 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:52.900 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:06:52.900 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:52.900 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:52.900 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:06:52.900 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:06:56.877 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:06:59.337 7f57811c80f0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:07:01.881 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:07:01.881 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:07:01.881 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:01.881 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:07:01.882 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:01.883 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:07:01.883 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:07:01.884 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:01.884 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:07:01.885 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:01.885 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:01.885 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:07:01.885 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:05.885 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:07:10.892 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:07:10.893 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:07:10.893 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:10.893 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:07:10.893 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:10.893 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:07:10.893 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:07:10.894 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:10.894 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:07:10.894 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:10.894 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:10.894 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:07:10.894 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:12.499 7f57875f0a20 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:07:14.895 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:07:19.903 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:07:19.918 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:07:19.919 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:19.920 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:07:19.920 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:19.923 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:07:19.925 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:07:19.925 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:19.925 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:07:19.925 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:19.925 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:19.925 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:07:19.925 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:23.906 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:07:28.915 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:07:28.917 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:07:28.917 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:28.917 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:07:28.917 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:28.917 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:07:28.917 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:07:28.917 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:28.918 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:07:28.918 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:28.918 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:28.918 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:07:28.918 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:32.917 7f577e2e98d0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:07:35.081 7f577d8bb4e0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:07:37.922 7f577e2e98d0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:07:37.923 7f577e2e98d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:07:37.923 7f577e2e98d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:37.923 7f577e2e98d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:07:37.923 7f577e2e98d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:37.923 7f577e2e98d0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:07:37.924 7f577e2e98d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:07:37.924 7f577e2e98d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:37.924 7f577e2e98d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:07:37.924 7f577e2e98d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:37.924 7f577e2e98d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:37.925 7f577e2e98d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:07:37.925 7f577e2e98d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:41.927 7f57875d5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:07:46.930 7f57875d5e10 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:07:46.931 7f57875d5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:07:46.931 7f57875d5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:46.931 7f57875d5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:07:46.931 7f57875d5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:46.931 7f57875d5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:07:46.932 7f57875d5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:07:46.932 7f57875d5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:46.932 7f57875d5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:07:46.933 7f57875d5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:46.933 7f57875d5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:46.933 7f57875d5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:07:46.933 7f57875d5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:50.931 7f577ead0240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:07:52.487 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51256364, partition_space_available_1k=29834120, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=26520, cpu_share=0.01, mem_res=10060, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=28408, cpu_share=0.02, mem_res=10104, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42128, cpu_share=0.0, mem_res=37184, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41872, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=14867232, free=10657136, buffers=984488, cached=6156264, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=22.3, five_min_avg=8.88, fifteen_min_avg=4.46, cpu_share=10.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:07:55.938 7f577ead0240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:07:55.939 7f577ead0240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:07:55.940 7f577ead0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:55.940 7f577ead0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:07:55.940 7f577ead0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:55.940 7f577ead0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:55.941 7f577ead0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51256364 partition_space_available_1k = 29834120 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 26520 cpu_share = 0.01 mem_res = 10060 >>contrail-dns : << mem_virt = 28408 cpu_share = 0.02 mem_res = 10104 >>contrail-named : << mem_virt = 42128 cpu_share = 0.0 mem_res = 37184 >>contrail-control-nodemgr : << mem_virt = 41872 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 14867232 free = 10657136 buffers = 984488 cached = 6156264 node_type = control-node >> system_cpu_usage = << one_min_avg = 22.3 five_min_avg = 8.88 fifteen_min_avg = 4.46 cpu_share = 10.38 node_type = control-node >> >> 06/02/2025 02:07:55.941 7f577ead0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:07:55.941 7f577ead0240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:07:55.942 7f577ead0240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:07:55.942 7f577ead0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:55.942 7f577ead0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:07:55.943 7f577ead0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:55.943 7f577ead0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:55.943 7f577ead0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:07:55.943 7f577ead0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:07:59.941 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:08:04.962 7f577d8a0828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:08:04.962 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:08:04.962 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:04.962 7f577d8a0828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:08:04.962 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:04.963 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:08:04.963 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:08:04.963 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:04.964 7f577d8a0828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:08:04.964 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:04.964 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:04.964 7f577d8a0828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:08:04.964 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:08.965 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:08:09.527 7f577d8c24e0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:08:13.972 7f577d8a0828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:08:13.972 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:08:13.972 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:13.972 7f577d8a0828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:08:13.973 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:13.973 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:08:13.973 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:08:13.973 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:13.973 7f577d8a0828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:08:13.974 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:13.974 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:13.974 7f577d8a0828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:08:13.974 7f577d8a0828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:17.973 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:08:22.979 7f577e2e9780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:08:22.979 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:08:22.979 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:22.979 7f577e2e9780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:08:22.980 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:22.980 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:08:22.981 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:08:22.981 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:22.981 7f577e2e9780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:08:22.981 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:22.981 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:22.981 7f577e2e9780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:08:22.981 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:24.741 7f577e2e9780 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:08:26.981 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:08:31.983 7f577e2e9780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:08:31.984 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:08:31.984 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:31.984 7f577e2e9780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:08:31.984 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:31.984 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:08:31.984 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:08:31.985 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:31.985 7f577e2e9780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:08:31.985 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:31.985 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:31.985 7f577e2e9780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:08:31.985 7f577e2e9780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:35.987 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:08:40.994 7f577e303978 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:08:40.994 7f577e303978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:08:40.994 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:40.994 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:08:40.994 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:40.994 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:08:40.995 7f577e303978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:08:40.995 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:40.995 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:08:40.995 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:40.995 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:40.995 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:08:40.995 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:44.995 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:08:46.543 7f577d88da20 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:08:49.999 7f577e303978 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:08:50.000 7f577e303978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:08:50.000 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:50.000 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:08:50.000 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:50.001 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:08:50.001 7f577e303978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:08:50.001 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:50.002 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:08:50.002 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:50.002 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:50.002 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:08:50.002 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:52.487 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51252592, partition_space_available_1k=29837892, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=31324, cpu_share=0.04, mem_res=13552, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=30736, cpu_share=0.04, mem_res=12420, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41876, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15289284, free=11193204, buffers=990316, cached=5192316, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=13.5, five_min_avg=8.83, fifteen_min_avg=4.72, cpu_share=5.88, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:08:52.487 7f57811e0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:52.488 7f57811e0d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51252592 partition_space_available_1k = 29837892 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 31324 cpu_share = 0.04 mem_res = 13552 >>contrail-dns : << mem_virt = 30736 cpu_share = 0.04 mem_res = 12420 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 41876 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15289284 free = 11193204 buffers = 990316 cached = 5192316 node_type = control-node >> system_cpu_usage = << one_min_avg = 13.5 five_min_avg = 8.83 fifteen_min_avg = 4.72 cpu_share = 5.88 node_type = control-node >> >> 06/02/2025 02:08:52.488 7f57811e0d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:54.001 7f57811e0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:08:56.993 7f577e2f0f60 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:08:59.005 7f57811e0d68 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:08:59.005 7f57811e0d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:08:59.005 7f57811e0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:59.005 7f57811e0d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:08:59.005 7f57811e0d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:08:59.005 7f57811e0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:08:59.006 7f57811e0d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:08:59.006 7f57811e0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:59.006 7f57811e0d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:08:59.006 7f57811e0d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:59.006 7f57811e0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:08:59.006 7f57811e0d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:08:59.006 7f57811e0d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:03.006 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:09:08.012 7f5781355ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:09:08.013 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:09:08.013 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:08.013 7f5781355ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:09:08.013 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:08.013 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:09:08.014 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:09:08.014 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:08.014 7f5781355ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:09:08.014 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:08.014 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:08.014 7f5781355ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:09:08.014 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:12.017 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:09:17.021 7f5781355ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:09:17.021 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:09:17.022 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:17.022 7f5781355ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:09:17.022 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:17.022 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:09:17.023 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:09:17.023 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:17.023 7f5781355ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:09:17.023 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:17.023 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:17.023 7f5781355ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:09:17.023 7f5781355ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:21.025 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:09:26.029 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:09:26.029 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:09:26.029 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:26.029 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:09:26.029 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:26.029 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:09:26.030 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:09:26.030 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:26.030 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:09:26.030 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:26.030 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:26.030 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:09:26.030 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:30.032 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:09:35.037 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:09:35.037 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:09:35.037 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:35.038 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:09:35.038 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:35.038 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:09:35.038 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:09:35.038 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:35.038 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:09:35.038 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:35.039 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:35.039 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:09:35.039 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:39.042 7f57824c1f60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:09:44.048 7f57824c1f60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:09:44.048 7f57824c1f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:09:44.049 7f57824c1f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:44.049 7f57824c1f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:09:44.049 7f57824c1f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:44.049 7f57824c1f60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:09:44.049 7f57824c1f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:09:44.049 7f57824c1f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:44.049 7f57824c1f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:09:44.050 7f57824c1f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:44.050 7f57824c1f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:44.050 7f57824c1f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:09:44.050 7f57824c1f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:48.049 7f577d8b6198 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:09:52.486 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51301080, partition_space_available_1k=29789404, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=31956, cpu_share=0.01, mem_res=14000, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=31232, cpu_share=0.01, mem_res=12848, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41936, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15093632, free=11267988, buffers=990872, cached=5312628, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=7.8, five_min_avg=8.1, fifteen_min_avg=4.73, cpu_share=4.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:09:53.051 7f577d8b6198 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:09:53.052 7f577d8b6198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:09:53.052 7f577d8b6198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:53.052 7f577d8b6198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:09:53.052 7f577d8b6198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:53.052 7f577d8b6198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:53.052 7f577d8b6198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51301080 partition_space_available_1k = 29789404 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 31956 cpu_share = 0.01 mem_res = 14000 >>contrail-dns : << mem_virt = 31232 cpu_share = 0.01 mem_res = 12848 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 41936 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15093632 free = 11267988 buffers = 990872 cached = 5312628 node_type = control-node >> system_cpu_usage = << one_min_avg = 7.8 five_min_avg = 8.1 fifteen_min_avg = 4.73 cpu_share = 4.38 node_type = control-node >> >> 06/02/2025 02:09:53.052 7f577d8b6198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:09:53.053 7f577d8b6198 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:09:53.053 7f577d8b6198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:09:53.054 7f577d8b6198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:53.054 7f577d8b6198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:09:53.054 7f577d8b6198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:53.054 7f577d8b6198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:53.054 7f577d8b6198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:09:53.054 7f577d8b6198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:09:57.055 7f57811a9828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:10:02.061 7f57811a9828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:10:02.061 7f57811a9828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:10:02.061 7f57811a9828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:02.061 7f57811a9828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:10:02.061 7f57811a9828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:02.062 7f57811a9828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:10:02.062 7f57811a9828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:10:02.062 7f57811a9828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:02.062 7f57811a9828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:10:02.062 7f57811a9828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:02.062 7f57811a9828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:02.063 7f57811a9828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:10:02.063 7f57811a9828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:06.063 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:10:11.069 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:10:11.069 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:10:11.069 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:11.069 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:10:11.069 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:11.069 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:10:11.070 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:10:11.070 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:11.070 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:10:11.070 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:11.070 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:11.070 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:10:11.070 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:15.073 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:10:17.958 7f577eacd198 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:10:20.077 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:10:20.078 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:10:20.078 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:20.078 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:10:20.078 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:20.078 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:10:20.078 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:10:20.079 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:20.079 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:10:20.079 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:20.079 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:20.079 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:10:20.079 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:24.081 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:10:28.655 7f5782844d68 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:10:29.083 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:10:29.083 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:10:29.084 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:29.084 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:10:29.084 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:29.084 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:10:29.084 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:10:29.084 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:29.085 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:10:29.085 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:29.085 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:29.085 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:10:29.085 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:33.083 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:10:38.089 7f57811a9cc0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:10:38.089 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:10:38.089 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:38.089 7f57811a9cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:10:38.089 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:38.089 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:10:38.090 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:10:38.090 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:38.090 7f57811a9cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:10:38.090 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:38.090 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:38.090 7f57811a9cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:10:38.090 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:42.093 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:10:47.095 7f57811a9cc0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:10:47.095 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:10:47.095 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:47.095 7f57811a9cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:10:47.096 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:47.096 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:10:47.096 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:10:47.096 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:47.096 7f57811a9cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:10:47.096 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:47.096 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:47.097 7f57811a9cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:10:47.097 7f57811a9cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:51.096 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:10:52.485 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51293268, partition_space_available_1k=29797216, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=32320, cpu_share=0.01, mem_res=14232, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=31540, cpu_share=0.01, mem_res=13096, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41960, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15139084, free=11118116, buffers=991144, cached=5416776, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.11, five_min_avg=6.96, fifteen_min_avg=4.55, cpu_share=3.12, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:10:56.101 7f57811e0240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:10:56.101 7f57811e0240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:10:56.101 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:56.101 7f57811e0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:10:56.101 7f57811e0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:56.102 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:56.102 7f57811e0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51293268 partition_space_available_1k = 29797216 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 32320 cpu_share = 0.01 mem_res = 14232 >>contrail-dns : << mem_virt = 31540 cpu_share = 0.01 mem_res = 13096 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 41960 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15139084 free = 11118116 buffers = 991144 cached = 5416776 node_type = control-node >> system_cpu_usage = << one_min_avg = 4.11 five_min_avg = 6.96 fifteen_min_avg = 4.55 cpu_share = 3.12 node_type = control-node >> >> 06/02/2025 02:10:56.102 7f57811e0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:10:56.102 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:10:56.103 7f57811e0240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:10:56.103 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:56.103 7f57811e0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:10:56.103 7f57811e0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:56.103 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:56.104 7f57811e0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:10:56.104 7f57811e0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:10:59.796 7f57811e0240 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:11:00.101 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:11:05.105 7f57811e0240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:11:05.105 7f57811e0240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:11:05.105 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:05.105 7f57811e0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:11:05.105 7f57811e0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:05.105 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:11:05.106 7f57811e0240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:11:05.106 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:05.106 7f57811e0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:11:05.106 7f57811e0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:05.106 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:05.106 7f57811e0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:11:05.107 7f57811e0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:09.108 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:11:14.115 7f57811e0240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:11:14.115 7f57811e0240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:11:14.115 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:14.115 7f57811e0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:11:14.116 7f57811e0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:14.116 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:11:14.116 7f57811e0240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:11:14.117 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:14.117 7f57811e0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:11:14.117 7f57811e0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:14.117 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:14.117 7f57811e0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:11:14.117 7f57811e0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:18.116 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:11:23.125 7f57811e0240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:11:23.125 7f57811e0240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:11:23.125 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:23.125 7f57811e0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:11:23.125 7f57811e0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:23.125 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:11:23.126 7f57811e0240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:11:23.126 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:23.126 7f57811e0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:11:23.126 7f57811e0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:23.126 7f57811e0240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:23.126 7f57811e0240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:11:23.126 7f57811e0240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:27.128 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:11:32.135 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:11:32.135 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:11:32.135 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:32.135 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:11:32.136 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:32.136 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:11:32.136 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:11:32.136 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:32.136 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:11:32.136 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:32.136 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:32.136 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:11:32.137 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:36.137 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:11:41.144 7f577e303978 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:11:41.144 7f577e303978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:11:41.144 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:41.144 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:11:41.144 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:41.144 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:11:41.145 7f577e303978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:11:41.145 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:41.145 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:11:41.145 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:41.145 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:41.145 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:11:41.145 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:45.145 7f577eac44e0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:11:50.148 7f577eac44e0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:11:50.148 7f577eac44e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:11:50.148 7f577eac44e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:50.148 7f577eac44e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:11:50.148 7f577eac44e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:50.148 7f577eac44e0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:11:50.149 7f577eac44e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:11:50.149 7f577eac44e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:50.149 7f577eac44e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:11:50.149 7f577eac44e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:50.149 7f577eac44e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:50.150 7f577eac44e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:11:50.150 7f577eac44e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:50.163 7f577eace8d0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:11:52.497 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51300644, partition_space_available_1k=29789840, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=32636, cpu_share=0.01, mem_res=14408, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=31772, cpu_share=0.01, mem_res=13264, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41964, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15167532, free=11043640, buffers=991976, cached=5461972, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.99, five_min_avg=6.44, fifteen_min_avg=4.52, cpu_share=2.88, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:11:52.497 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:52.497 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51300644 partition_space_available_1k = 29789840 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 32636 cpu_share = 0.01 mem_res = 14408 >>contrail-dns : << mem_virt = 31772 cpu_share = 0.01 mem_res = 13264 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 41964 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15167532 free = 11043640 buffers = 991976 cached = 5461972 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.99 five_min_avg = 6.44 fifteen_min_avg = 4.52 cpu_share = 2.88 node_type = control-node >> >> 06/02/2025 02:11:52.497 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:54.150 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:11:59.157 7f577e303978 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:11:59.157 7f577e303978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:11:59.157 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:59.157 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:11:59.157 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:11:59.157 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:11:59.158 7f577e303978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:11:59.158 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:59.158 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:11:59.158 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:59.158 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:11:59.158 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:11:59.158 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:03.161 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:12:08.165 7f577e303978 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:12:08.165 7f577e303978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:12:08.165 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:08.165 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:12:08.166 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:08.166 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:12:08.166 7f577e303978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:12:08.166 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:08.166 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:12:08.166 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:08.166 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:08.167 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:12:08.167 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:12.173 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:12:17.181 7f577e303978 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:12:17.181 7f577e303978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:12:17.181 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:17.181 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:12:17.182 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:17.182 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:12:17.182 7f577e303978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:12:17.183 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:17.184 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:12:17.184 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:17.184 7f577e303978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:17.184 7f577e303978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:12:17.184 7f577e303978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:21.183 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:12:26.190 7f57811cccc0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:12:26.190 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:12:26.190 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:26.190 7f57811cccc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:12:26.190 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:26.190 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:12:26.190 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:12:26.191 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:26.191 7f57811cccc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:12:26.191 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:26.191 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:26.191 7f57811cccc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:12:26.191 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:30.194 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:12:35.196 7f57811cccc0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:12:35.197 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:12:35.197 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:35.197 7f57811cccc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:12:35.197 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:35.197 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:12:35.197 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:12:35.198 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:35.198 7f57811cccc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:12:35.198 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:35.198 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:35.198 7f57811cccc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:12:35.198 7f57811cccc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:39.198 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:12:44.205 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:12:44.205 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:12:44.205 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:44.205 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:12:44.205 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:44.205 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:12:44.206 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:12:44.206 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:44.206 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:12:44.206 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:44.206 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:44.206 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:12:44.206 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:48.207 7f577d8c28d0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:12:52.483 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51313664, partition_space_available_1k=29776820, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=32732, cpu_share=0.01, mem_res=14376, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=31956, cpu_share=0.01, mem_res=13388, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42020, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15134560, free=9741100, buffers=991988, cached=6797472, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.35, five_min_avg=6.81, fifteen_min_avg=4.77, cpu_share=5.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:12:53.210 7f577d8c28d0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:12:53.211 7f577d8c28d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:12:53.211 7f577d8c28d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:53.211 7f577d8c28d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:12:53.211 7f577d8c28d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:53.211 7f577d8c28d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:53.211 7f577d8c28d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51313664 partition_space_available_1k = 29776820 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 32732 cpu_share = 0.01 mem_res = 14376 >>contrail-dns : << mem_virt = 31956 cpu_share = 0.01 mem_res = 13388 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42020 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15134560 free = 9741100 buffers = 991988 cached = 6797472 node_type = control-node >> system_cpu_usage = << one_min_avg = 6.35 five_min_avg = 6.81 fifteen_min_avg = 4.77 cpu_share = 5.25 node_type = control-node >> >> 06/02/2025 02:12:53.211 7f577d8c28d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:12:53.211 7f577d8c28d0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:12:53.212 7f577d8c28d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:12:53.212 7f577d8c28d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:53.212 7f577d8c28d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:12:53.212 7f577d8c28d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:53.212 7f577d8c28d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:53.212 7f577d8c28d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:12:53.212 7f577d8c28d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:12:57.214 7f577e308240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:13:02.221 7f577e308240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:13:02.221 7f577e308240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:13:02.221 7f577e308240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:02.221 7f577e308240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:13:02.221 7f577e308240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:02.221 7f577e308240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:13:02.222 7f577e308240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:13:02.222 7f577e308240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:02.222 7f577e308240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:13:02.222 7f577e308240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:02.222 7f577e308240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:02.222 7f577e308240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:13:02.222 7f577e308240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:06.223 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:13:11.229 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:13:11.229 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:13:11.229 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:11.229 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:13:11.229 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:11.229 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:13:11.230 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:13:11.230 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:11.230 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:13:11.230 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:11.230 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:11.230 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:13:11.230 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:15.233 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:13:20.236 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:13:20.236 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:13:20.236 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:20.237 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:13:20.237 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:20.237 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:13:20.237 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:13:20.237 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:20.237 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:13:20.237 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:20.237 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:20.238 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:13:20.238 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:24.238 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:13:29.244 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:13:29.245 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:13:29.245 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:29.245 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:13:29.245 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:29.245 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:13:29.245 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:13:29.245 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:29.245 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:13:29.245 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:29.245 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:29.246 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:13:29.246 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:33.249 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:13:38.253 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:13:38.254 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:13:38.254 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:38.254 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:13:38.254 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:38.254 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:13:38.255 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:13:38.255 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:38.255 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:13:38.255 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:38.255 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:38.256 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:13:38.256 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:42.258 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:13:47.259 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:13:47.260 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:13:47.260 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:47.260 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:13:47.260 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:47.260 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:13:47.261 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:13:47.261 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:47.261 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:13:47.261 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:47.261 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:47.261 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:13:47.261 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:51.260 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:13:52.492 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51320204, partition_space_available_1k=29770280, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33168, cpu_share=0.01, mem_res=14676, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=32152, cpu_share=0.01, mem_res=13520, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42040, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15140500, free=9727616, buffers=991992, cached=6805012, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.5, five_min_avg=6.15, fifteen_min_avg=4.67, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:13:56.265 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:13:56.266 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:13:56.266 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:56.266 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:13:56.266 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:56.266 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:56.266 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51320204 partition_space_available_1k = 29770280 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 33168 cpu_share = 0.01 mem_res = 14676 >>contrail-dns : << mem_virt = 32152 cpu_share = 0.01 mem_res = 13520 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42040 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15140500 free = 9727616 buffers = 991992 cached = 6805012 node_type = control-node >> system_cpu_usage = << one_min_avg = 4.5 five_min_avg = 6.15 fifteen_min_avg = 4.67 cpu_share = 2.62 node_type = control-node >> >> 06/02/2025 02:13:56.266 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:13:56.266 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:13:56.267 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:13:56.267 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:56.267 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:13:56.267 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:56.267 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:13:56.267 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:13:56.267 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:00.269 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:14:05.271 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:14:05.271 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:14:05.272 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:05.272 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:14:05.272 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:05.272 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:14:05.272 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:14:05.273 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:05.273 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:14:05.273 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:05.273 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:05.273 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:14:05.273 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:09.275 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:14:14.280 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:14:14.281 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:14:14.281 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:14.281 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:14:14.281 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:14.281 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:14:14.282 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:14:14.282 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:14.282 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:14:14.282 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:14.282 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:14.282 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:14:14.282 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:18.285 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:14:23.289 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:14:23.289 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:14:23.289 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:23.289 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:14:23.289 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:23.289 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:14:23.290 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:14:23.290 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:23.290 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:14:23.290 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:23.290 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:23.290 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:14:23.290 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:27.293 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:14:32.300 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:14:32.300 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:14:32.300 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:32.300 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:14:32.301 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:32.301 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:14:32.302 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:14:32.302 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:32.302 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:14:32.302 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:32.302 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:32.302 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:14:32.302 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:36.301 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:14:41.306 7f577d8c2240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:14:41.306 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:14:41.306 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:41.306 7f577d8c2240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:14:41.306 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:41.306 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:14:41.307 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:14:41.307 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:41.307 7f577d8c2240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:14:41.307 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:41.307 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:41.307 7f577d8c2240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:14:41.307 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:45.307 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:14:50.311 7f577d8c2240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:14:50.312 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:14:50.312 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:50.312 7f577d8c2240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:14:50.312 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:50.313 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:14:50.313 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:14:50.313 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:50.314 7f577d8c2240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:14:50.314 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:50.314 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:50.314 7f577d8c2240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:14:50.314 7f577d8c2240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:52.490 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51326052, partition_space_available_1k=29764432, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33288, cpu_share=0.01, mem_res=14664, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=32368, cpu_share=0.01, mem_res=13676, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42032, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15167884, free=9688264, buffers=992000, cached=6816972, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.71, five_min_avg=5.33, fifteen_min_avg=4.48, cpu_share=2.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:14:52.490 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:52.490 7f577d8c2978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51326052 partition_space_available_1k = 29764432 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 33288 cpu_share = 0.01 mem_res = 14664 >>contrail-dns : << mem_virt = 32368 cpu_share = 0.01 mem_res = 13676 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42032 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15167884 free = 9688264 buffers = 992000 cached = 6816972 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.71 five_min_avg = 5.33 fifteen_min_avg = 4.48 cpu_share = 2.75 node_type = control-node >> >> 06/02/2025 02:14:52.490 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:54.312 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:14:59.317 7f577d8c2978 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:14:59.318 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:14:59.318 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:59.318 7f577d8c2978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:14:59.318 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:14:59.318 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:14:59.318 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:14:59.319 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:59.319 7f577d8c2978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:14:59.319 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:59.319 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:14:59.319 7f577d8c2978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:14:59.319 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:03.322 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:15:04.207 7f57811c8828 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:15:08.325 7f577d8c2978 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:15:08.325 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:15:08.325 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:08.325 7f577d8c2978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:15:08.325 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:08.325 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:15:08.326 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:15:08.326 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:08.326 7f577d8c2978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:15:08.326 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:08.326 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:08.326 7f577d8c2978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:15:08.326 7f577d8c2978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:12.329 7f57811bd438 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:15:14.536 7f577e308b70 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:15:17.334 7f57811bd438 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:15:17.335 7f57811bd438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:15:17.335 7f57811bd438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:17.335 7f57811bd438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:15:17.335 7f57811bd438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:17.335 7f57811bd438 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:15:17.336 7f57811bd438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:15:17.337 7f57811bd438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:17.337 7f57811bd438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:15:17.337 7f57811bd438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:17.337 7f57811bd438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:17.337 7f57811bd438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:15:17.337 7f57811bd438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:21.335 7f57811cc048 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:15:25.275 7f57811c7240 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:15:26.337 7f57811cc048 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:15:26.338 7f57811cc048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:15:26.338 7f57811cc048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:26.338 7f57811cc048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:15:26.338 7f57811cc048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:26.338 7f57811cc048 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:15:26.339 7f57811cc048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:15:26.339 7f57811cc048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:26.339 7f57811cc048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:15:26.339 7f57811cc048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:26.339 7f57811cc048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:26.339 7f57811cc048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:15:26.339 7f57811cc048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:30.341 7f57811cc048 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:15:35.347 7f57811cc048 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:15:35.347 7f57811cc048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:15:35.347 7f57811cc048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:35.348 7f57811cc048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:15:35.348 7f57811cc048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:35.348 7f57811cc048 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:15:35.349 7f57811cc048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:15:35.350 7f57811cc048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:35.350 7f57811cc048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:15:35.351 7f57811cc048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:35.351 7f57811cc048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:35.351 7f57811cc048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:15:35.351 7f57811cc048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:39.350 7f577eaffa20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:15:44.353 7f577eaffa20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:15:44.353 7f577eaffa20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:15:44.353 7f577eaffa20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:44.353 7f577eaffa20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:15:44.353 7f577eaffa20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:44.353 7f577eaffa20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:15:44.354 7f577eaffa20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:15:44.354 7f577eaffa20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:44.354 7f577eaffa20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:15:44.354 7f577eaffa20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:44.354 7f577eaffa20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:44.355 7f577eaffa20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:15:44.355 7f577eaffa20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:48.354 7f57811e0588 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:15:52.494 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51335540, partition_space_available_1k=29754944, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33632, cpu_share=0.01, mem_res=14872, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=32472, cpu_share=0.01, mem_res=13716, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42092, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15141984, free=9700204, buffers=993360, cached=6829572, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.91, five_min_avg=4.92, fifteen_min_avg=4.39, cpu_share=3.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:15:53.357 7f57811e0588 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:15:53.357 7f57811e0588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:15:53.357 7f57811e0588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:53.357 7f57811e0588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:15:53.357 7f57811e0588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:53.357 7f57811e0588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:53.358 7f57811e0588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51335540 partition_space_available_1k = 29754944 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 33632 cpu_share = 0.01 mem_res = 14872 >>contrail-dns : << mem_virt = 32472 cpu_share = 0.01 mem_res = 13716 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42092 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15141984 free = 9700204 buffers = 993360 cached = 6829572 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.91 five_min_avg = 4.92 fifteen_min_avg = 4.39 cpu_share = 3.38 node_type = control-node >> >> 06/02/2025 02:15:53.358 7f57811e0588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:15:53.358 7f57811e0588 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:15:53.358 7f57811e0588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:15:53.358 7f57811e0588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:53.358 7f57811e0588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:15:53.358 7f57811e0588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:53.358 7f57811e0588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:53.359 7f57811e0588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:15:53.359 7f57811e0588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:15:57.361 7f577e2f0b70 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:16:02.373 7f577e2f0b70 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:16:02.373 7f577e2f0b70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:16:02.374 7f577e2f0b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:02.374 7f577e2f0b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:16:02.374 7f577e2f0b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:02.374 7f577e2f0b70 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:16:02.374 7f577e2f0b70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:16:02.374 7f577e2f0b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:02.374 7f577e2f0b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:16:02.375 7f577e2f0b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:02.375 7f577e2f0b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:02.375 7f577e2f0b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:16:02.375 7f577e2f0b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:06.373 7f57810560f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:16:11.382 7f57810560f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:16:11.382 7f57810560f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:16:11.382 7f57810560f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:11.383 7f57810560f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:16:11.383 7f57810560f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:11.383 7f57810560f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:16:11.383 7f57810560f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:16:11.383 7f57810560f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:11.383 7f57810560f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:16:11.383 7f57810560f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:11.383 7f57810560f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:11.384 7f57810560f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:16:11.384 7f57810560f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:15.385 7f57810560f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:16:20.391 7f57810560f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:16:20.391 7f57810560f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:16:20.391 7f57810560f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:20.391 7f57810560f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:16:20.392 7f57810560f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:20.392 7f57810560f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:16:20.392 7f57810560f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:16:20.392 7f57810560f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:20.392 7f57810560f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:16:20.392 7f57810560f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:20.392 7f57810560f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:20.393 7f57810560f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:16:20.393 7f57810560f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:24.396 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:16:29.401 7f577d8b6eb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:16:29.401 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:16:29.401 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:29.401 7f577d8b6eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:16:29.401 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:29.401 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:16:29.402 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:16:29.402 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:29.402 7f577d8b6eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:16:29.402 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:29.402 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:29.402 7f577d8b6eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:16:29.403 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:33.405 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:16:38.409 7f577d8b6eb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:16:38.409 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:16:38.409 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:38.409 7f577d8b6eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:16:38.409 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:38.409 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:16:38.410 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:16:38.410 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:38.410 7f577d8b6eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:16:38.410 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:38.410 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:38.411 7f577d8b6eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:16:38.411 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:42.413 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:16:47.416 7f577d8b6eb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:16:47.417 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:16:47.417 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:47.417 7f577d8b6eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:16:47.418 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:47.418 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:16:47.418 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:16:47.418 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:47.419 7f577d8b6eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:16:47.419 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:47.419 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:47.419 7f577d8b6eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:16:47.419 7f577d8b6eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:48.918 7f57811aeb70 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:16:51.417 7f577d8b6438 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:16:52.521 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51346996, partition_space_available_1k=29743488, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33900, cpu_share=0.01, mem_res=15004, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=32692, cpu_share=0.01, mem_res=13868, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42116, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15251700, free=9554956, buffers=994020, cached=6864444, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.5, five_min_avg=5.44, fifteen_min_avg=4.59, cpu_share=5.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:16:56.423 7f577d8b6438 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:16:56.424 7f577d8b6438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:16:56.424 7f577d8b6438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:56.424 7f577d8b6438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:16:56.424 7f577d8b6438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:56.424 7f577d8b6438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:56.424 7f577d8b6438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51346996 partition_space_available_1k = 29743488 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 33900 cpu_share = 0.01 mem_res = 15004 >>contrail-dns : << mem_virt = 32692 cpu_share = 0.01 mem_res = 13868 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42116 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15251700 free = 9554956 buffers = 994020 cached = 6864444 node_type = control-node >> system_cpu_usage = << one_min_avg = 6.5 five_min_avg = 5.44 fifteen_min_avg = 4.59 cpu_share = 5.25 node_type = control-node >> >> 06/02/2025 02:16:56.424 7f577d8b6438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:16:56.424 7f577d8b6438 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:16:56.425 7f577d8b6438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:16:56.425 7f577d8b6438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:56.425 7f577d8b6438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:16:56.425 7f577d8b6438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:56.425 7f577d8b6438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:16:56.425 7f577d8b6438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:16:56.425 7f577d8b6438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:00.424 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:17:05.431 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:17:05.431 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:17:05.431 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:05.432 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:17:05.432 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:05.432 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:17:05.432 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:17:05.433 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:05.433 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:17:05.433 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:05.433 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:05.433 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:17:05.433 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:09.433 7f57811a9630 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:17:14.437 7f57811a9630 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:17:14.437 7f57811a9630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:17:14.437 7f57811a9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:14.437 7f57811a9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:17:14.437 7f57811a9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:14.437 7f57811a9630 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:17:14.438 7f57811a9630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:17:14.438 7f57811a9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:14.438 7f57811a9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:17:14.438 7f57811a9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:14.438 7f57811a9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:14.439 7f57811a9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:17:14.439 7f57811a9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:18.441 7f57811a9630 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:17:23.445 7f57811a9630 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:17:23.445 7f57811a9630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:17:23.445 7f57811a9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:23.445 7f57811a9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:17:23.445 7f57811a9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:23.445 7f57811a9630 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:17:23.446 7f57811a9630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:17:23.446 7f57811a9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:23.446 7f57811a9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:17:23.446 7f57811a9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:23.446 7f57811a9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:23.446 7f57811a9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:17:23.446 7f57811a9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:27.449 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:17:32.456 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:17:32.456 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:17:32.456 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:32.456 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:17:32.456 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:32.456 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:17:32.457 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:17:32.457 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:32.457 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:17:32.457 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:32.457 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:32.457 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:17:32.457 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:36.460 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:17:41.465 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:17:41.466 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:17:41.466 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:41.466 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:17:41.466 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:41.466 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:17:41.467 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:17:41.467 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:41.467 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:17:41.467 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:41.467 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:41.467 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:17:41.467 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:45.466 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:17:50.469 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:17:50.469 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:17:50.469 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:50.469 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:17:50.470 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:50.470 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:17:50.470 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:17:50.470 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:50.470 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:17:50.470 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:50.470 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:50.470 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:17:50.471 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:52.515 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51355292, partition_space_available_1k=29735192, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34192, cpu_share=0.01, mem_res=15164, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=32828, cpu_share=0.01, mem_res=13940, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42112, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15206124, free=9578432, buffers=994116, cached=6886448, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.66, five_min_avg=4.79, fifteen_min_avg=4.42, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:17:52.515 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:52.515 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51355292 partition_space_available_1k = 29735192 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 34192 cpu_share = 0.01 mem_res = 15164 >>contrail-dns : << mem_virt = 32828 cpu_share = 0.01 mem_res = 13940 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42112 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15206124 free = 9578432 buffers = 994116 cached = 6886448 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.66 five_min_avg = 4.79 fifteen_min_avg = 4.42 cpu_share = 2.62 node_type = control-node >> >> 06/02/2025 02:17:52.515 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:54.471 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:17:59.477 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:17:59.477 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:17:59.477 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:59.477 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:17:59.478 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:17:59.478 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:17:59.478 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:17:59.479 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:59.479 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:17:59.479 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:59.479 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:17:59.479 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:17:59.479 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:03.481 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:18:08.484 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:18:08.484 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:18:08.484 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:08.484 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:18:08.484 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:08.484 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:18:08.485 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:18:08.485 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:08.485 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:18:08.485 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:08.485 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:08.485 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:18:08.485 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:12.489 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:18:17.493 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:18:17.493 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:18:17.493 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:17.493 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:18:17.493 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:17.493 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:18:17.494 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:18:17.494 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:17.494 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:18:17.494 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:17.494 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:17.494 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:18:17.495 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:21.497 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:18:26.502 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:18:26.502 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:18:26.502 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:26.502 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:18:26.502 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:26.503 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:18:26.503 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:18:26.503 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:26.503 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:18:26.503 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:26.503 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:26.503 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:18:26.503 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:30.505 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:18:35.509 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:18:35.509 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:18:35.509 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:35.509 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:18:35.509 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:35.509 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:18:35.509 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:18:35.510 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:35.510 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:18:35.510 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:35.510 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:35.510 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:18:35.510 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:39.510 7f577d8bbc18 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:18:44.512 7f577d8bbc18 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:18:44.513 7f577d8bbc18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:18:44.513 7f577d8bbc18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:44.513 7f577d8bbc18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:18:44.513 7f577d8bbc18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:44.513 7f577d8bbc18 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:18:44.514 7f577d8bbc18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:18:44.515 7f577d8bbc18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:44.515 7f577d8bbc18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:18:44.515 7f577d8bbc18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:44.515 7f577d8bbc18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:44.515 7f577d8bbc18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:18:44.515 7f577d8bbc18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:48.512 7f57811aeac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:18:52.500 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51360800, partition_space_available_1k=29729684, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34584, cpu_share=0.01, mem_res=15420, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33016, cpu_share=0.01, mem_res=14064, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42160, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15200248, free=9577392, buffers=994116, cached=6893364, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.52, five_min_avg=4.26, fifteen_min_avg=4.26, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:18:53.514 7f57811aeac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:18:53.515 7f57811aeac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:18:53.515 7f57811aeac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:53.515 7f57811aeac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:18:53.515 7f57811aeac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:53.515 7f57811aeac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:53.516 7f57811aeac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51360800 partition_space_available_1k = 29729684 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 34584 cpu_share = 0.01 mem_res = 15420 >>contrail-dns : << mem_virt = 33016 cpu_share = 0.01 mem_res = 14064 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42160 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15200248 free = 9577392 buffers = 994116 cached = 6893364 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.52 five_min_avg = 4.26 fifteen_min_avg = 4.26 cpu_share = 2.62 node_type = control-node >> >> 06/02/2025 02:18:53.516 7f57811aeac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:18:53.516 7f57811aeac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:18:53.517 7f57811aeac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:18:53.517 7f57811aeac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:53.517 7f57811aeac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:18:53.517 7f57811aeac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:53.517 7f57811aeac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:53.517 7f57811aeac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:18:53.517 7f57811aeac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:18:57.518 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:19:02.525 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:19:02.525 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:19:02.525 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:02.525 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:19:02.525 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:02.525 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:19:02.526 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:19:02.526 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:02.526 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:19:02.526 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:02.526 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:02.526 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:19:02.526 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:06.529 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:19:11.534 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:19:11.534 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:19:11.534 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:11.535 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:19:11.535 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:11.535 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:19:11.535 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:19:11.535 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:11.536 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:19:11.536 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:11.536 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:11.536 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:19:11.536 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:15.538 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:19:20.544 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:19:20.544 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:19:20.544 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:20.544 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:19:20.545 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:20.545 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:19:20.545 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:19:20.545 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:20.546 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:19:20.546 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:20.546 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:20.546 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:19:20.546 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:24.545 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:19:29.551 7f577eaffcc0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:19:29.552 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:19:29.552 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:29.552 7f577eaffcc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:19:29.552 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:29.552 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:19:29.553 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:19:29.553 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:29.553 7f577eaffcc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:19:29.553 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:29.553 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:29.553 7f577eaffcc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:19:29.553 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:33.553 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:19:38.555 7f577eaffcc0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:19:38.555 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:19:38.555 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:38.556 7f577eaffcc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:19:38.556 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:38.556 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:19:38.557 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:19:38.557 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:38.557 7f577eaffcc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:19:38.557 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:38.557 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:38.557 7f577eaffcc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:19:38.557 7f577eaffcc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:42.559 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:19:47.562 7f577e2f0978 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:19:47.562 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:19:47.562 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:47.563 7f577e2f0978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:19:47.563 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:47.563 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:19:47.563 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:19:47.563 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:47.563 7f577e2f0978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:19:47.564 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:47.564 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:47.564 7f577e2f0978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:19:47.564 7f577e2f0978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:51.563 7f577d741b70 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:19:52.506 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51366132, partition_space_available_1k=29724352, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34620, cpu_share=0.01, mem_res=15324, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33168, cpu_share=0.01, mem_res=14160, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42180, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15185364, free=9586320, buffers=994124, cached=6899312, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.33, five_min_avg=3.91, fifteen_min_avg=4.14, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:19:56.566 7f577d741b70 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:19:56.566 7f577d741b70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:19:56.566 7f577d741b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:56.566 7f577d741b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:19:56.566 7f577d741b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:56.566 7f577d741b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:56.566 7f577d741b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51366132 partition_space_available_1k = 29724352 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 34620 cpu_share = 0.01 mem_res = 15324 >>contrail-dns : << mem_virt = 33168 cpu_share = 0.01 mem_res = 14160 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42180 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15185364 free = 9586320 buffers = 994124 cached = 6899312 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.33 five_min_avg = 3.91 fifteen_min_avg = 4.14 cpu_share = 2.62 node_type = control-node >> >> 06/02/2025 02:19:56.566 7f577d741b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:19:56.567 7f577d741b70 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:19:56.567 7f577d741b70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:19:56.567 7f577d741b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:56.567 7f577d741b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:19:56.567 7f577d741b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:56.567 7f577d741b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:19:56.568 7f577d741b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:19:56.568 7f577d741b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:00.569 7f577d8bba20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:20:05.575 7f577d8bba20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:20:05.575 7f577d8bba20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:20:05.575 7f577d8bba20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:05.575 7f577d8bba20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:20:05.575 7f577d8bba20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:05.575 7f577d8bba20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:20:05.576 7f577d8bba20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:20:05.576 7f577d8bba20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:05.576 7f577d8bba20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:20:05.576 7f577d8bba20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:05.576 7f577d8bba20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:05.576 7f577d8bba20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:20:05.576 7f577d8bba20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:09.576 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:20:14.582 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:20:14.582 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:20:14.582 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:14.583 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:20:14.583 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:14.583 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:20:14.584 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:20:14.584 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:14.584 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:20:14.584 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:14.584 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:14.584 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:20:14.585 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:16.616 7f5781361eb8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:20:18.584 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:20:23.590 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:20:23.590 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:20:23.590 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:23.591 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:20:23.591 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:23.591 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:20:23.591 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:20:23.591 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:23.591 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:20:23.591 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:23.591 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:23.592 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:20:23.592 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:27.329 7f5782462ac8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:20:27.590 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:20:32.597 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:20:32.597 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:20:32.597 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:32.597 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:20:32.598 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:32.598 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:20:32.598 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:20:32.598 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:32.598 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:20:32.598 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:32.598 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:32.599 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:20:32.599 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:36.601 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:20:37.767 7f577eaff828 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:20:41.605 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:20:41.606 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:20:41.606 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:41.606 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:20:41.606 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:41.606 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:20:41.607 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:20:41.607 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:41.607 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:20:41.607 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:41.607 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:41.607 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:20:41.607 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:45.607 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:20:50.609 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:20:50.609 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:20:50.609 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:50.609 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:20:50.609 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:50.609 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:20:50.610 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:20:50.610 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:50.610 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:20:50.610 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:50.610 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:50.610 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:20:50.610 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:52.514 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51375696, partition_space_available_1k=29714788, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34860, cpu_share=0.01, mem_res=15428, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33256, cpu_share=0.01, mem_res=14184, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42172, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15240432, free=9518232, buffers=995508, cached=6910948, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.31, five_min_avg=3.61, fifteen_min_avg=4.02, cpu_share=3.12, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:20:52.514 7f57811c5198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:52.514 7f57811c5198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51375696 partition_space_available_1k = 29714788 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 34860 cpu_share = 0.01 mem_res = 15428 >>contrail-dns : << mem_virt = 33256 cpu_share = 0.01 mem_res = 14184 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42172 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15240432 free = 9518232 buffers = 995508 cached = 6910948 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.31 five_min_avg = 3.61 fifteen_min_avg = 4.02 cpu_share = 3.12 node_type = control-node >> >> 06/02/2025 02:20:52.514 7f57811c5198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:54.609 7f577d8a0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:20:59.615 7f577d8a0f60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:20:59.615 7f577d8a0f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:20:59.616 7f577d8a0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:59.616 7f577d8a0f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:20:59.616 7f577d8a0f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:20:59.616 7f577d8a0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:20:59.616 7f577d8a0f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:20:59.616 7f577d8a0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:59.616 7f577d8a0f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:20:59.617 7f577d8a0f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:59.617 7f577d8a0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:20:59.617 7f577d8a0f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:20:59.617 7f577d8a0f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:03.620 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:21:08.626 7f5780cb3390 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:21:08.626 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:21:08.626 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:08.626 7f5780cb3390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:21:08.626 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:08.626 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:21:08.627 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:21:08.627 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:08.627 7f5780cb3390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:21:08.627 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:08.627 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:08.627 7f5780cb3390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:21:08.627 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:12.629 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:21:17.635 7f577eac46d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:21:17.636 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:21:17.636 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:17.636 7f577eac46d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:21:17.636 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:17.636 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:21:17.637 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:21:17.637 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:17.637 7f577eac46d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:21:17.637 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:17.637 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:17.638 7f577eac46d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:21:17.638 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:21.639 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:21:26.642 7f577eac46d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:21:26.642 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:21:26.642 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:26.643 7f577eac46d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:21:26.643 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:26.643 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:21:26.643 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:21:26.644 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:26.644 7f577eac46d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:21:26.644 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:26.645 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:26.645 7f577eac46d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:21:26.645 7f577eac46d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:30.646 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:21:35.652 7f577e2f06d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:21:35.653 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:21:35.653 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:35.653 7f577e2f06d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:21:35.653 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:35.653 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:21:35.653 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:21:35.654 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:35.654 7f577e2f06d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:21:35.654 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:35.654 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:35.654 7f577e2f06d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:21:35.654 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:39.657 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:21:44.658 7f577e2f06d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:21:44.658 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:21:44.659 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:44.659 7f577e2f06d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:21:44.659 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:44.659 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:21:44.660 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:21:44.660 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:44.660 7f577e2f06d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:21:44.660 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:44.660 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:44.660 7f577e2f06d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:21:44.660 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:46.302 7f577e2f06d8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:21:48.658 7f577ead3240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:21:52.518 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51382932, partition_space_available_1k=29707552, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35048, cpu_share=0.01, mem_res=15480, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33352, cpu_share=0.01, mem_res=14220, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42224, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15292832, free=9442952, buffers=996092, cached=6933244, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.25, five_min_avg=4.46, fifteen_min_avg=4.28, cpu_share=5.12, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:21:53.661 7f577ead3240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:21:53.661 7f577ead3240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:21:53.662 7f577ead3240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:53.662 7f577ead3240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:21:53.662 7f577ead3240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:53.662 7f577ead3240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:53.662 7f577ead3240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51382932 partition_space_available_1k = 29707552 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 35048 cpu_share = 0.01 mem_res = 15480 >>contrail-dns : << mem_virt = 33352 cpu_share = 0.01 mem_res = 14220 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42224 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15292832 free = 9442952 buffers = 996092 cached = 6933244 node_type = control-node >> system_cpu_usage = << one_min_avg = 6.25 five_min_avg = 4.46 fifteen_min_avg = 4.28 cpu_share = 5.12 node_type = control-node >> >> 06/02/2025 02:21:53.662 7f577ead3240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:21:53.662 7f577ead3240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:21:53.663 7f577ead3240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:21:53.663 7f577ead3240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:53.663 7f577ead3240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:21:53.663 7f577ead3240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:53.663 7f577ead3240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:53.663 7f577ead3240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:21:53.663 7f577ead3240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:21:57.663 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:22:02.668 7f57811c8cc0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:22:02.669 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:22:02.669 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:02.669 7f57811c8cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:22:02.669 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:02.669 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:22:02.670 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:22:02.670 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:02.670 7f57811c8cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:22:02.670 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:02.670 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:02.670 7f57811c8cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:22:02.670 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:06.673 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:22:11.676 7f57811c8cc0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:22:11.676 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:22:11.676 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:11.677 7f57811c8cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:22:11.677 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:11.677 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:22:11.677 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:22:11.677 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:11.677 7f57811c8cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:22:11.677 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:11.677 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:11.678 7f57811c8cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:22:11.678 7f57811c8cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:15.678 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:22:20.685 7f5787002240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:22:20.685 7f5787002240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:22:20.685 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:20.685 7f5787002240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:22:20.685 7f5787002240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:20.685 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:22:20.686 7f5787002240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:22:20.686 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:20.686 7f5787002240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:22:20.686 7f5787002240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:20.686 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:20.686 7f5787002240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:22:20.687 7f5787002240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:24.689 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:22:29.696 7f5787002240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:22:29.696 7f5787002240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:22:29.696 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:29.697 7f5787002240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:22:29.697 7f5787002240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:29.697 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:22:29.697 7f5787002240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:22:29.697 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:29.697 7f5787002240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:22:29.697 7f5787002240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:29.697 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:29.697 7f5787002240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:22:29.698 7f5787002240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:33.699 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:22:38.705 7f5787002240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:22:38.705 7f5787002240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:22:38.705 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:38.705 7f5787002240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:22:38.705 7f5787002240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:38.705 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:22:38.706 7f5787002240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:22:38.706 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:38.706 7f5787002240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:22:38.706 7f5787002240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:38.706 7f5787002240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:38.706 7f5787002240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:22:38.707 7f5787002240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:42.707 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:22:47.709 7f577ec35828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:22:47.710 7f577ec35828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:22:47.710 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:47.710 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:22:47.710 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:47.710 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:22:47.711 7f577ec35828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:22:47.711 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:47.711 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:22:47.711 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:47.711 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:47.711 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:22:47.711 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:51.710 7f57811a9ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:22:52.506 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51390768, partition_space_available_1k=29699716, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35120, cpu_share=0.01, mem_res=15420, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33524, cpu_share=0.01, mem_res=14328, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42252, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15261380, free=9469100, buffers=996100, cached=6938540, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.69, five_min_avg=4.06, fifteen_min_avg=4.15, cpu_share=2.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:22:56.713 7f57811a9ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:22:56.713 7f57811a9ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:22:56.713 7f57811a9ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:56.713 7f57811a9ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:22:56.713 7f57811a9ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:56.713 7f57811a9ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:56.714 7f57811a9ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51390768 partition_space_available_1k = 29699716 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 35120 cpu_share = 0.01 mem_res = 15420 >>contrail-dns : << mem_virt = 33524 cpu_share = 0.01 mem_res = 14328 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42252 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15261380 free = 9469100 buffers = 996100 cached = 6938540 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.69 five_min_avg = 4.06 fifteen_min_avg = 4.15 cpu_share = 2.38 node_type = control-node >> >> 06/02/2025 02:22:56.714 7f57811a9ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:22:56.714 7f57811a9ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:22:56.714 7f57811a9ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:22:56.714 7f57811a9ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:56.714 7f57811a9ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:22:56.714 7f57811a9ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:56.714 7f57811a9ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:22:56.715 7f57811a9ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:22:56.715 7f57811a9ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:00.717 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:23:05.723 7f577e2e9828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:23:05.723 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:23:05.723 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:05.723 7f577e2e9828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:23:05.723 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:05.723 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:23:05.724 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:23:05.724 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:05.724 7f577e2e9828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:23:05.724 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:05.724 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:05.724 7f577e2e9828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:23:05.724 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:09.726 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:23:14.731 7f577e2e9828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:23:14.731 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:23:14.731 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:14.731 7f577e2e9828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:23:14.731 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:14.732 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:23:14.732 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:23:14.732 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:14.732 7f577e2e9828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:23:14.732 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:14.732 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:14.733 7f577e2e9828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:23:14.733 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:18.733 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:23:23.739 7f577e2e9828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:23:23.740 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:23:23.740 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:23.740 7f577e2e9828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:23:23.740 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:23.740 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:23:23.741 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:23:23.741 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:23.742 7f577e2e9828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:23:23.742 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:23.742 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:23.742 7f577e2e9828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:23:23.742 7f577e2e9828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:27.741 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:23:32.746 7f57811c8ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:23:32.747 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:23:32.747 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:32.747 7f57811c8ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:23:32.747 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:32.747 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:23:32.748 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:23:32.748 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:32.748 7f57811c8ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:23:32.748 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:32.748 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:32.748 7f57811c8ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:23:32.748 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:36.749 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:23:41.751 7f57811c8ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:23:41.751 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:23:41.751 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:41.751 7f57811c8ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:23:41.751 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:41.751 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:23:41.752 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:23:41.752 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:41.752 7f57811c8ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:23:41.752 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:41.752 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:41.752 7f57811c8ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:23:41.752 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:45.751 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:23:50.754 7f57811c8ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:23:50.754 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:23:50.754 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:50.754 7f57811c8ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:23:50.755 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:50.755 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:23:50.755 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:23:50.755 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:50.755 7f57811c8ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:23:50.755 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:50.755 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:50.755 7f57811c8ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:23:50.756 7f57811c8ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:52.513 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51396116, partition_space_available_1k=29694368, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35428, cpu_share=0.01, mem_res=15592, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33596, cpu_share=0.01, mem_res=14336, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42244, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15266480, free=9462452, buffers=996100, cached=6940088, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.4, five_min_avg=3.86, fifteen_min_avg=4.08, cpu_share=2.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:23:52.514 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:52.514 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51396116 partition_space_available_1k = 29694368 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 35428 cpu_share = 0.01 mem_res = 15592 >>contrail-dns : << mem_virt = 33596 cpu_share = 0.01 mem_res = 14336 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42244 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15266480 free = 9462452 buffers = 996100 cached = 6940088 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.4 five_min_avg = 3.86 fifteen_min_avg = 4.08 cpu_share = 2.25 node_type = control-node >> >> 06/02/2025 02:23:52.514 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:54.756 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:23:59.760 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:23:59.760 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:23:59.760 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:59.761 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:23:59.761 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:23:59.761 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:23:59.761 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:23:59.761 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:59.762 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:23:59.762 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:59.762 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:23:59.762 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:23:59.762 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:03.762 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:24:08.769 7f57811610f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:24:08.769 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:24:08.769 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:08.770 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:24:08.770 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:08.770 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:24:08.770 7f57811610f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:24:08.770 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:08.771 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:24:08.771 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:08.771 7f57811610f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:08.771 7f57811610f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:24:08.771 7f57811610f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:12.770 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:24:17.776 7f577d8a0048 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:24:17.776 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:24:17.776 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:17.776 7f577d8a0048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:24:17.776 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:17.777 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:24:17.777 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:24:17.777 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:17.777 7f577d8a0048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:24:17.777 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:17.777 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:17.777 7f577d8a0048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:24:17.777 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:21.780 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:24:26.782 7f577d8a0048 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:24:26.782 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:24:26.782 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:26.782 7f577d8a0048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:24:26.782 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:26.782 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:24:26.783 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:24:26.783 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:26.784 7f577d8a0048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:24:26.784 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:26.784 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:26.784 7f577d8a0048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:24:26.784 7f577d8a0048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:30.786 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:24:35.792 7f577cb7c6d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:24:35.793 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:24:35.793 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:35.793 7f577cb7c6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:24:35.793 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:35.793 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:24:35.794 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:24:35.794 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:35.794 7f577cb7c6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:24:35.794 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:35.794 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:35.794 7f577cb7c6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:24:35.795 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:39.795 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:24:44.797 7f577cb7c6d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:24:44.798 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:24:44.798 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:44.799 7f577cb7c6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:24:44.799 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:44.799 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:24:44.800 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:24:44.800 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:44.800 7f577cb7c6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:24:44.800 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:44.801 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:44.801 7f577cb7c6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:24:44.801 7f577cb7c6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:48.798 7f57811c8b70 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:24:52.518 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51396636, partition_space_available_1k=29693848, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35556, cpu_share=0.01, mem_res=15588, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33660, cpu_share=0.01, mem_res=14336, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42300, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15257608, free=9464884, buffers=996100, cached=6946528, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.13, five_min_avg=3.4, fifteen_min_avg=3.9, cpu_share=2.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:24:53.801 7f57811c8b70 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:24:53.801 7f57811c8b70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:24:53.801 7f57811c8b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:53.801 7f57811c8b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:24:53.802 7f57811c8b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:53.802 7f57811c8b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:53.802 7f57811c8b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51396636 partition_space_available_1k = 29693848 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 35556 cpu_share = 0.01 mem_res = 15588 >>contrail-dns : << mem_virt = 33660 cpu_share = 0.01 mem_res = 14336 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42300 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15257608 free = 9464884 buffers = 996100 cached = 6946528 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.13 five_min_avg = 3.4 fifteen_min_avg = 3.9 cpu_share = 2.25 node_type = control-node >> >> 06/02/2025 02:24:53.802 7f57811c8b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:24:53.802 7f57811c8b70 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:24:53.803 7f57811c8b70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:24:53.803 7f57811c8b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:53.803 7f57811c8b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:24:53.803 7f57811c8b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:53.803 7f57811c8b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:53.803 7f57811c8b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:24:53.803 7f57811c8b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:24:57.803 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:25:02.809 7f577e2f0f60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:25:02.809 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:25:02.809 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:02.809 7f577e2f0f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:25:02.809 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:02.809 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:25:02.810 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:25:02.810 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:02.810 7f577e2f0f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:25:02.810 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:02.810 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:02.810 7f577e2f0f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:25:02.811 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:06.813 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:25:11.818 7f577e2f0f60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:25:11.818 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:25:11.819 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:11.819 7f577e2f0f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:25:11.819 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:11.819 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:25:11.819 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:25:11.819 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:11.819 7f577e2f0f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:25:11.819 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:11.819 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:11.820 7f577e2f0f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:25:11.820 7f577e2f0f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:15.820 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:25:20.825 7f577d7414e0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:25:20.825 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:25:20.826 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:20.826 7f577d7414e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:25:20.826 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:20.826 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:25:20.826 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:25:20.826 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:20.826 7f577d7414e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:25:20.826 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:20.826 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:20.827 7f577d7414e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:25:20.827 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:24.828 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:25:29.832 7f577d7414e0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:25:29.833 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:25:29.833 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:29.833 7f577d7414e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:25:29.833 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:29.833 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:25:29.833 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:25:29.834 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:29.834 7f577d7414e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:25:29.834 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:29.834 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:29.834 7f577d7414e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:25:29.834 7f577d7414e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:33.837 7f577ec1feb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:25:38.842 7f577ec1feb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:25:38.843 7f577ec1feb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:25:38.843 7f577ec1feb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:38.843 7f577ec1feb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:25:38.843 7f577ec1feb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:38.843 7f577ec1feb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:25:38.843 7f577ec1feb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:25:38.843 7f577ec1feb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:38.844 7f577ec1feb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:25:38.844 7f577ec1feb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:38.844 7f577ec1feb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:38.844 7f577ec1feb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:25:38.844 7f577ec1feb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:42.843 7f5780c64588 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:25:47.847 7f5780c64588 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:25:47.847 7f5780c64588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:25:47.847 7f5780c64588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:47.848 7f5780c64588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:25:47.848 7f5780c64588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:47.848 7f5780c64588 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:25:47.849 7f5780c64588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:25:47.849 7f5780c64588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:47.849 7f5780c64588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:25:47.849 7f5780c64588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:47.849 7f5780c64588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:47.849 7f5780c64588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:25:47.849 7f5780c64588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:51.848 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:25:52.557 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51400664, partition_space_available_1k=29689820, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35592, cpu_share=0.01, mem_res=15488, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33720, cpu_share=0.01, mem_res=14336, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42316, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15359304, free=9348480, buffers=996100, cached=6961236, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.48, five_min_avg=3.93, fifteen_min_avg=4.04, cpu_share=4.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:25:56.854 7f57811c76d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:25:56.854 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:25:56.854 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:56.854 7f57811c76d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:25:56.854 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:56.855 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:56.855 7f57811c76d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51400664 partition_space_available_1k = 29689820 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 35592 cpu_share = 0.01 mem_res = 15488 >>contrail-dns : << mem_virt = 33720 cpu_share = 0.01 mem_res = 14336 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42316 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15359304 free = 9348480 buffers = 996100 cached = 6961236 node_type = control-node >> system_cpu_usage = << one_min_avg = 5.48 five_min_avg = 3.93 fifteen_min_avg = 4.04 cpu_share = 4.75 node_type = control-node >> >> 06/02/2025 02:25:56.855 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:25:56.855 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:25:56.856 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:25:56.856 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:56.856 7f57811c76d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:25:56.856 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:56.857 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:25:56.857 7f57811c76d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:25:56.857 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:00.857 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:26:05.862 7f57811cca20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:26:05.863 7f57811cca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:26:05.863 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:05.863 7f57811cca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:26:05.863 7f57811cca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:05.863 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:26:05.863 7f57811cca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:26:05.863 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:05.863 7f57811cca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:26:05.865 7f57811cca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:05.866 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:05.866 7f57811cca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:26:05.866 7f57811cca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:08.578 7f57811cca20 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:26:09.863 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:26:14.867 7f57811cca20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:26:14.868 7f57811cca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:26:14.868 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:14.868 7f57811cca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:26:14.868 7f57811cca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:14.868 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:26:14.869 7f57811cca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:26:14.869 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:14.869 7f57811cca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:26:14.869 7f57811cca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:14.869 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:14.869 7f57811cca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:26:14.869 7f57811cca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:18.867 7f577d8a0b70 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:26:18.869 7f577d8c2ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:26:23.875 7f577d8c2ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:26:23.876 7f577d8c2ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:26:23.876 7f577d8c2ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:23.876 7f577d8c2ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:26:23.876 7f577d8c2ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:23.876 7f577d8c2ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:26:23.877 7f577d8c2ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:26:23.877 7f577d8c2ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:23.877 7f577d8c2ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:26:23.877 7f577d8c2ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:23.877 7f577d8c2ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:23.877 7f577d8c2ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:26:23.877 7f577d8c2ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:27.876 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:26:29.410 7f577ead0e10 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:26:32.881 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:26:32.882 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:26:32.882 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:32.882 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:26:32.882 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:32.882 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:26:32.883 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:26:32.883 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:32.883 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:26:32.883 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:32.883 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:32.883 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:26:32.883 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:36.885 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:26:41.889 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:26:41.889 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:26:41.889 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:41.889 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:26:41.889 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:41.889 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:26:41.890 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:26:41.890 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:41.890 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:26:41.890 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:41.890 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:41.890 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:26:41.890 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:45.891 7f577d8b2f60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:26:50.894 7f577d8b2f60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:26:50.894 7f577d8b2f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:26:50.894 7f577d8b2f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:50.894 7f577d8b2f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:26:50.894 7f577d8b2f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:50.894 7f577d8b2f60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:26:50.895 7f577d8b2f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:26:50.895 7f577d8b2f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:50.895 7f577d8b2f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:26:50.895 7f577d8b2f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:50.895 7f577d8b2f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:50.895 7f577d8b2f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:26:50.896 7f577d8b2f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:52.525 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51409196, partition_space_available_1k=29681288, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35596, cpu_share=0.01, mem_res=15356, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33892, cpu_share=0.01, mem_res=14444, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42308, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15313740, free=9473160, buffers=996144, cached=6882076, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.28, five_min_avg=3.91, fifteen_min_avg=4.03, cpu_share=3.0, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:26:52.525 7f577d741780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:52.525 7f577d741780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51409196 partition_space_available_1k = 29681288 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 35596 cpu_share = 0.01 mem_res = 15356 >>contrail-dns : << mem_virt = 33892 cpu_share = 0.01 mem_res = 14444 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42308 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15313740 free = 9473160 buffers = 996144 cached = 6882076 node_type = control-node >> system_cpu_usage = << one_min_avg = 4.28 five_min_avg = 3.91 fifteen_min_avg = 4.03 cpu_share = 3.0 node_type = control-node >> >> 06/02/2025 02:26:52.525 7f577d741780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:54.897 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:26:59.898 7f5787250048 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:26:59.899 7f5787250048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:26:59.899 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:59.899 7f5787250048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:26:59.899 7f5787250048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:26:59.899 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:26:59.900 7f5787250048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:26:59.900 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:59.900 7f5787250048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:26:59.900 7f5787250048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:59.900 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:26:59.900 7f5787250048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:26:59.900 7f5787250048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:03.903 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:27:08.909 7f5787250048 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:27:08.909 7f5787250048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:27:08.909 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:08.909 7f5787250048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:27:08.910 7f5787250048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:08.910 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:27:08.910 7f5787250048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:27:08.910 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:08.910 7f5787250048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:27:08.910 7f5787250048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:08.910 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:08.911 7f5787250048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:27:08.911 7f5787250048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:12.913 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:27:17.919 7f5787250048 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:27:17.919 7f5787250048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:27:17.919 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:17.920 7f5787250048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:27:17.920 7f5787250048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:17.920 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:27:17.920 7f5787250048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:27:17.920 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:17.921 7f5787250048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:27:17.921 7f5787250048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:17.921 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:17.921 7f5787250048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:27:17.921 7f5787250048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:21.923 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:27:26.929 7f5787250048 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:27:26.929 7f5787250048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:27:26.929 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:26.930 7f5787250048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:27:26.930 7f5787250048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:26.930 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:27:26.930 7f5787250048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:27:26.930 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:26.930 7f5787250048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:27:26.930 7f5787250048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:26.930 7f5787250048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:26.930 7f5787250048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:27:26.930 7f5787250048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:30.931 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:27:33.706 7f577eacf2e8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:27:35.933 7f577d8b22e8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:27:35.934 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:27:35.934 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:35.934 7f577d8b22e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:27:35.934 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:35.934 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:27:35.935 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:27:35.935 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:35.936 7f577d8b22e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:27:35.936 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:35.936 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:35.936 7f577d8b22e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:27:35.936 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:39.938 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:27:44.940 7f577d8b22e8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:27:44.941 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:27:44.941 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:44.941 7f577d8b22e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:27:44.941 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:44.941 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:27:44.942 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:27:44.942 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:44.942 7f577d8b22e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:27:44.942 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:44.942 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:44.942 7f577d8b22e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:27:44.942 7f577d8b22e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:48.941 7f577eaff048 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:27:52.516 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51411420, partition_space_available_1k=29679064, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35824, cpu_share=0.01, mem_res=15452, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=33968, cpu_share=0.01, mem_res=14460, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42368, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15288380, free=9497940, buffers=996164, cached=6882636, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.84, five_min_avg=3.77, fifteen_min_avg=3.97, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:27:53.943 7f577eaff048 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:27:53.943 7f577eaff048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:27:53.944 7f577eaff048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:53.944 7f577eaff048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:27:53.944 7f577eaff048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:53.944 7f577eaff048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:53.945 7f577eaff048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51411420 partition_space_available_1k = 29679064 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 35824 cpu_share = 0.01 mem_res = 15452 >>contrail-dns : << mem_virt = 33968 cpu_share = 0.01 mem_res = 14460 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42368 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15288380 free = 9497940 buffers = 996164 cached = 6882636 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.84 five_min_avg = 3.77 fifteen_min_avg = 3.97 cpu_share = 2.62 node_type = control-node >> >> 06/02/2025 02:27:53.945 7f577eaff048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:27:53.945 7f577eaff048 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:27:53.945 7f577eaff048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:27:53.945 7f577eaff048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:53.945 7f577eaff048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:27:53.946 7f577eaff048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:53.946 7f577eaff048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:53.946 7f577eaff048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:27:53.946 7f577eaff048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:27:57.948 7f577e308438 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:28:02.953 7f577e308438 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:28:02.953 7f577e308438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:28:02.953 7f577e308438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:02.953 7f577e308438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:28:02.953 7f577e308438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:02.953 7f577e308438 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:28:02.954 7f577e308438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:28:02.954 7f577e308438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:02.954 7f577e308438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:28:02.954 7f577e308438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:02.954 7f577e308438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:02.954 7f577e308438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:28:02.954 7f577e308438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:06.956 7f577e308438 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:28:11.962 7f577e308438 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:28:11.962 7f577e308438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:28:11.962 7f577e308438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:11.962 7f577e308438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:28:11.962 7f577e308438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:11.962 7f577e308438 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:28:11.963 7f577e308438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:28:11.963 7f577e308438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:11.963 7f577e308438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:28:11.963 7f577e308438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:11.963 7f577e308438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:11.963 7f577e308438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:28:11.964 7f577e308438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:15.964 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:28:20.971 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:28:20.971 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:28:20.971 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:20.971 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:28:20.971 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:20.972 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:28:20.972 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:28:20.972 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:20.972 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:28:20.972 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:20.972 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:20.973 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:28:20.973 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:24.974 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:28:29.977 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:28:29.977 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:28:29.977 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:29.978 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:28:29.978 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:29.978 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:28:29.978 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:28:29.978 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:29.978 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:28:29.978 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:29.979 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:29.979 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:28:29.979 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:33.982 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:28:38.988 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:28:38.989 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:28:38.989 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:38.989 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:28:38.989 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:38.989 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:28:38.989 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:28:38.990 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:38.990 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:28:38.990 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:38.990 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:38.990 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:28:38.990 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:42.994 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:28:47.997 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:28:47.997 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:28:47.997 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:47.997 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:28:47.997 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:47.997 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:28:47.998 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:28:47.998 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:47.998 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:28:47.998 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:47.998 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:47.998 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:28:47.998 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:51.999 7f577d88dd68 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:28:52.517 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51417016, partition_space_available_1k=29673468, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=36028, cpu_share=0.01, mem_res=15520, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34028, cpu_share=0.01, mem_res=14460, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42392, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15300396, free=9484388, buffers=996164, cached=6884172, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.35, five_min_avg=3.35, fifteen_min_avg=3.81, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:28:57.005 7f577d88dd68 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:28:57.005 7f577d88dd68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:28:57.005 7f577d88dd68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:57.005 7f577d88dd68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:28:57.005 7f577d88dd68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:57.005 7f577d88dd68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:57.006 7f577d88dd68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51417016 partition_space_available_1k = 29673468 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 36028 cpu_share = 0.01 mem_res = 15520 >>contrail-dns : << mem_virt = 34028 cpu_share = 0.01 mem_res = 14460 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42392 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15300396 free = 9484388 buffers = 996164 cached = 6884172 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.35 five_min_avg = 3.35 fifteen_min_avg = 3.81 cpu_share = 2.62 node_type = control-node >> >> 06/02/2025 02:28:57.006 7f577d88dd68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:28:57.006 7f577d88dd68 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:28:57.006 7f577d88dd68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:28:57.006 7f577d88dd68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:57.006 7f577d88dd68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:28:57.006 7f577d88dd68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:57.006 7f577d88dd68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:28:57.007 7f577d88dd68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:28:57.007 7f577d88dd68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:01.006 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:29:06.013 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:29:06.013 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:29:06.013 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:06.013 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:29:06.013 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:06.013 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:29:06.014 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:29:06.014 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:06.014 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:29:06.014 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:06.014 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:06.014 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:29:06.014 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:10.018 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:29:15.020 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:29:15.021 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:29:15.021 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:15.021 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:29:15.021 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:15.021 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:29:15.022 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:29:15.022 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:15.022 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:29:15.022 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:15.022 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:15.022 7f577ead0c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:29:15.022 7f577ead0c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:19.024 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:29:24.031 7f57811a9f60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:29:24.031 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:29:24.031 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:24.031 7f57811a9f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:29:24.031 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:24.031 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:29:24.032 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:29:24.032 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:24.032 7f57811a9f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:29:24.032 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:24.032 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:24.032 7f57811a9f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:29:24.032 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:28.035 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:29:33.041 7f57811a9f60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:29:33.041 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:29:33.041 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:33.041 7f57811a9f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:29:33.041 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:33.042 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:29:33.042 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:29:33.042 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:33.042 7f57811a9f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:29:33.042 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:33.042 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:33.042 7f57811a9f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:29:33.042 7f57811a9f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:37.044 7f57811aef60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:29:42.051 7f57811aef60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:29:42.051 7f57811aef60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:29:42.051 7f57811aef60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:42.051 7f57811aef60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:29:42.051 7f57811aef60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:42.051 7f57811aef60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:29:42.052 7f57811aef60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:29:42.052 7f57811aef60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:42.052 7f57811aef60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:29:42.052 7f57811aef60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:42.052 7f57811aef60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:42.052 7f57811aef60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:29:42.052 7f57811aef60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:46.053 7f577cd61588 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:29:51.055 7f577cd61588 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:29:51.055 7f577cd61588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:29:51.056 7f577cd61588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:51.057 7f577cd61588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:29:51.057 7f577cd61588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:29:51.057 7f577cd61588 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:29:51.057 7f577cd61588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:29:51.057 7f577cd61588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:51.058 7f577cd61588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:29:51.058 7f577cd61588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:51.058 7f577cd61588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:51.058 7f577cd61588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:29:51.059 7f577cd61588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:52.518 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51417492, partition_space_available_1k=29672992, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=36416, cpu_share=0.01, mem_res=15776, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34100, cpu_share=0.01, mem_res=14468, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42380, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15282840, free=9495724, buffers=996180, cached=6890376, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.19, five_min_avg=3.14, fifteen_min_avg=3.71, cpu_share=2.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:29:52.518 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:52.518 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51417492 partition_space_available_1k = 29672992 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 36416 cpu_share = 0.01 mem_res = 15776 >>contrail-dns : << mem_virt = 34100 cpu_share = 0.01 mem_res = 14468 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42380 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15282840 free = 9495724 buffers = 996180 cached = 6890376 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.19 five_min_avg = 3.14 fifteen_min_avg = 3.71 cpu_share = 2.5 node_type = control-node >> >> 06/02/2025 02:29:52.519 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:29:55.058 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:30:00.060 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:30:00.060 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:30:00.060 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:00.060 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:30:00.060 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:00.060 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:30:00.061 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:30:00.061 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:00.061 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:30:00.061 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:00.061 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:00.061 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:30:00.061 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:04.062 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:30:09.069 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:30:09.069 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:30:09.069 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:09.069 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:30:09.069 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:09.069 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:30:09.070 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:30:09.070 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:09.070 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:30:09.070 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:09.070 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:09.070 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:30:09.070 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:13.073 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:30:18.077 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:30:18.077 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:30:18.077 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:18.077 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:30:18.078 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:18.078 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:30:18.078 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:30:18.078 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:18.078 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:30:18.078 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:18.078 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:18.079 7f577eaff6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:30:18.079 7f577eaff6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:22.082 7f577e308a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:30:27.087 7f577e308a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:30:27.087 7f577e308a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:30:27.087 7f577e308a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:27.088 7f577e308a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:30:27.088 7f577e308a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:27.088 7f577e308a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:30:27.089 7f577e308a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:30:27.089 7f577e308a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:27.089 7f577e308a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:30:27.089 7f577e308a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:27.089 7f577e308a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:27.089 7f577e308a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:30:27.089 7f577e308a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:31.088 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:30:36.093 7f57811c76d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:30:36.093 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:30:36.093 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:36.093 7f57811c76d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:30:36.093 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:36.093 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:30:36.094 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:30:36.094 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:36.094 7f57811c76d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:30:36.094 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:36.094 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:36.094 7f57811c76d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:30:36.094 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:40.097 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:30:45.099 7f57811c76d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:30:45.099 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:30:45.100 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:45.100 7f57811c76d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:30:45.100 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:45.100 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:30:45.101 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:30:45.101 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:45.101 7f57811c76d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:30:45.101 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:45.101 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:45.101 7f57811c76d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:30:45.101 7f57811c76d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:49.100 7f577eace240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:30:52.364 7f577eacd0f0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:30:52.535 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51423332, partition_space_available_1k=29667152, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=36596, cpu_share=0.01, mem_res=15820, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34160, cpu_share=0.01, mem_res=14468, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42436, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15403320, free=9332924, buffers=997792, cached=6931084, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=6.1, five_min_avg=3.97, fifteen_min_avg=3.95, cpu_share=5.0, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:30:54.106 7f577eace240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:30:54.106 7f577eace240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:30:54.106 7f577eace240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:54.106 7f577eace240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:30:54.107 7f577eace240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:54.107 7f577eace240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:54.107 7f577eace240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51423332 partition_space_available_1k = 29667152 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 36596 cpu_share = 0.01 mem_res = 15820 >>contrail-dns : << mem_virt = 34160 cpu_share = 0.01 mem_res = 14468 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42436 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15403320 free = 9332924 buffers = 997792 cached = 6931084 node_type = control-node >> system_cpu_usage = << one_min_avg = 6.1 five_min_avg = 3.97 fifteen_min_avg = 3.95 cpu_share = 5.0 node_type = control-node >> >> 06/02/2025 02:30:54.107 7f577eace240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:30:54.107 7f577eace240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:30:54.107 7f577eace240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:30:54.107 7f577eace240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:54.108 7f577eace240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:30:54.108 7f577eace240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:54.108 7f577eace240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:54.108 7f577eace240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:30:54.108 7f577eace240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:30:58.110 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:31:02.766 7f57811c74e0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:31:03.112 7f5780cd3978 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:31:03.113 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:31:03.113 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:03.113 7f5780cd3978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:31:03.113 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:03.113 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:31:03.113 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:31:03.113 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:03.113 7f5780cd3978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:31:03.113 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:03.114 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:03.114 7f5780cd3978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:31:03.114 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:07.116 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:31:12.122 7f5780cd3978 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:31:12.122 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:31:12.122 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:12.122 7f5780cd3978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:31:12.123 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:12.123 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:31:12.123 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:31:12.123 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:12.123 7f5780cd3978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:31:12.123 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:12.124 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:12.124 7f5780cd3978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:31:12.124 7f5780cd3978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:13.323 7f5780cd3978 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:31:16.123 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:31:21.129 7f57811cca20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:31:21.129 7f57811cca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:31:21.130 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:21.130 7f57811cca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:31:21.130 7f57811cca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:21.130 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:31:21.130 7f57811cca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:31:21.130 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:21.130 7f57811cca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:31:21.131 7f57811cca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:21.131 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:21.131 7f57811cca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:31:21.131 7f57811cca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:25.133 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:31:30.138 7f57811cca20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:31:30.139 7f57811cca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:31:30.139 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:30.139 7f57811cca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:31:30.139 7f57811cca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:30.139 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:31:30.139 7f57811cca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:31:30.140 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:30.140 7f57811cca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:31:30.140 7f57811cca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:30.140 7f57811cca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:30.140 7f57811cca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:31:30.140 7f57811cca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:34.143 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:31:39.149 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:31:39.149 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:31:39.149 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:39.149 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:31:39.149 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:39.149 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:31:39.150 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:31:39.150 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:39.150 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:31:39.150 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:39.150 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:39.150 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:31:39.150 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:43.153 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:31:48.156 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:31:48.156 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:31:48.156 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:48.157 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:31:48.157 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:48.157 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:31:48.157 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:31:48.157 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:48.158 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:31:48.158 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:48.158 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:48.158 7f5782462ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:31:48.158 7f5782462ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:52.157 7f57811e0b70 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:31:52.537 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51428968, partition_space_available_1k=29661516, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=36868, cpu_share=0.01, mem_res=15956, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34224, cpu_share=0.01, mem_res=14468, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42456, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15312648, free=9422516, buffers=997792, cached=6932164, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.52, five_min_avg=3.65, fifteen_min_avg=3.84, cpu_share=2.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:31:57.161 7f57811e0b70 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:31:57.161 7f57811e0b70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:31:57.161 7f57811e0b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:57.162 7f57811e0b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:31:57.162 7f57811e0b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:57.162 7f57811e0b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:57.162 7f57811e0b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51428968 partition_space_available_1k = 29661516 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 36868 cpu_share = 0.01 mem_res = 15956 >>contrail-dns : << mem_virt = 34224 cpu_share = 0.01 mem_res = 14468 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42456 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15312648 free = 9422516 buffers = 997792 cached = 6932164 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.52 five_min_avg = 3.65 fifteen_min_avg = 3.84 cpu_share = 2.75 node_type = control-node >> >> 06/02/2025 02:31:57.162 7f57811e0b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:31:57.162 7f57811e0b70 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:31:57.163 7f57811e0b70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:31:57.163 7f57811e0b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:57.163 7f57811e0b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:31:57.163 7f57811e0b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:57.163 7f57811e0b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:31:57.163 7f57811e0b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:31:57.163 7f57811e0b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:01.162 7f57813614e0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:32:06.169 7f57813614e0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:32:06.169 7f57813614e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:32:06.169 7f57813614e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:06.169 7f57813614e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:32:06.169 7f57813614e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:06.169 7f57813614e0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:32:06.170 7f57813614e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:32:06.170 7f57813614e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:06.170 7f57813614e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:32:06.170 7f57813614e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:06.170 7f57813614e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:06.170 7f57813614e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:32:06.170 7f57813614e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:10.172 7f57813614e0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:32:15.179 7f57813614e0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:32:15.179 7f57813614e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:32:15.179 7f57813614e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:15.179 7f57813614e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:32:15.180 7f57813614e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:15.180 7f57813614e0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:32:15.180 7f57813614e0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:32:15.180 7f57813614e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:15.180 7f57813614e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:32:15.180 7f57813614e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:15.180 7f57813614e0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:15.180 7f57813614e0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:32:15.180 7f57813614e0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:19.182 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:32:24.188 7f577d7412e8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:32:24.188 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:32:24.188 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:24.189 7f577d7412e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:32:24.189 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:24.189 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:32:24.189 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:32:24.190 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:24.190 7f577d7412e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:32:24.190 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:24.190 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:24.190 7f577d7412e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:32:24.190 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:28.192 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:32:30.310 7f57811e0d68 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:32:33.195 7f577d7412e8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:32:33.195 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:32:33.196 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:33.196 7f577d7412e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:32:33.196 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:33.196 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:32:33.196 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:32:33.196 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:33.197 7f577d7412e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:32:33.197 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:33.197 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:33.197 7f577d7412e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:32:33.197 7f577d7412e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:37.200 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:32:42.205 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:32:42.205 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:32:42.205 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:42.206 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:32:42.206 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:42.206 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:32:42.206 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:32:42.206 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:42.206 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:32:42.206 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:42.206 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:42.206 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:32:42.207 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:46.208 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:32:51.211 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:32:51.212 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:32:51.212 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:51.212 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:32:51.212 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:32:51.212 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:32:51.213 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:32:51.213 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:51.213 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:32:51.213 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:51.213 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:51.214 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:32:51.214 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:52.536 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51430932, partition_space_available_1k=29659552, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=36912, cpu_share=0.01, mem_res=15868, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34368, cpu_share=0.01, mem_res=14552, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42448, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15333944, free=9398504, buffers=997792, cached=6934880, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.93, five_min_avg=3.42, fifteen_min_avg=3.75, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:32:52.536 7f57811e0eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:52.536 7f57811e0eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51430932 partition_space_available_1k = 29659552 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 36912 cpu_share = 0.01 mem_res = 15868 >>contrail-dns : << mem_virt = 34368 cpu_share = 0.01 mem_res = 14552 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42448 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15333944 free = 9398504 buffers = 997792 cached = 6934880 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.93 five_min_avg = 3.42 fifteen_min_avg = 3.75 cpu_share = 2.62 node_type = control-node >> >> 06/02/2025 02:32:52.536 7f57811e0eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:32:55.214 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:33:00.221 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:33:00.221 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:33:00.221 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:00.221 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:33:00.221 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:00.221 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:33:00.222 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:33:00.222 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:00.222 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:33:00.222 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:00.222 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:00.222 7f5781137ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:33:00.222 7f5781137ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:04.224 7f57811c8198 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:33:09.230 7f57811c8198 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:33:09.230 7f57811c8198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:33:09.230 7f57811c8198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:09.231 7f57811c8198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:33:09.231 7f57811c8198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:09.231 7f57811c8198 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:33:09.232 7f57811c8198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:33:09.232 7f57811c8198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:09.232 7f57811c8198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:33:09.232 7f57811c8198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:09.232 7f57811c8198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:09.232 7f57811c8198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:33:09.232 7f57811c8198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:13.234 7f57811c8198 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:33:18.237 7f57811c8198 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:33:18.237 7f57811c8198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:33:18.237 7f57811c8198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:18.237 7f57811c8198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:33:18.237 7f57811c8198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:18.237 7f57811c8198 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:33:18.238 7f57811c8198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:33:18.238 7f57811c8198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:18.238 7f57811c8198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:33:18.238 7f57811c8198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:18.238 7f57811c8198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:18.238 7f57811c8198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:33:18.238 7f57811c8198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:22.241 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:33:27.248 7f57811e0e10 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:33:27.248 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:33:27.248 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:27.248 7f57811e0e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:33:27.248 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:27.248 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:33:27.249 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:33:27.249 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:27.249 7f57811e0e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:33:27.250 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:27.250 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:27.250 7f57811e0e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:33:27.250 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:31.253 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:33:36.259 7f57811e0e10 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:33:36.260 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:33:36.260 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:36.260 7f57811e0e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:33:36.260 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:36.260 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:33:36.261 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:33:36.261 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:36.261 7f57811e0e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:33:36.261 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:36.261 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:36.261 7f57811e0e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:33:36.261 7f57811e0e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:40.263 7f577cd56828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:33:45.266 7f577cd56828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:33:45.266 7f577cd56828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:33:45.266 7f577cd56828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:45.267 7f577cd56828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:33:45.267 7f577cd56828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:45.267 7f577cd56828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:33:45.267 7f577cd56828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:33:45.268 7f577cd56828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:45.268 7f577cd56828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:33:45.268 7f577cd56828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:45.268 7f577cd56828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:45.268 7f577cd56828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:33:45.268 7f577cd56828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:49.267 7f577eaced68 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:33:52.542 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51436720, partition_space_available_1k=29653764, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=37012, cpu_share=0.01, mem_res=15832, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34448, cpu_share=0.01, mem_res=14564, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42500, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15306048, free=9419728, buffers=997792, cached=6941552, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.32, five_min_avg=3.13, fifteen_min_avg=3.62, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:33:54.270 7f577eaced68 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:33:54.271 7f577eaced68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:33:54.271 7f577eaced68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:54.271 7f577eaced68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:33:54.271 7f577eaced68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:54.271 7f577eaced68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:54.271 7f577eaced68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51436720 partition_space_available_1k = 29653764 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 37012 cpu_share = 0.01 mem_res = 15832 >>contrail-dns : << mem_virt = 34448 cpu_share = 0.01 mem_res = 14564 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42500 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15306048 free = 9419728 buffers = 997792 cached = 6941552 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.32 five_min_avg = 3.13 fifteen_min_avg = 3.62 cpu_share = 2.62 node_type = control-node >> >> 06/02/2025 02:33:54.271 7f577eaced68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:33:54.272 7f577eaced68 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:33:54.272 7f577eaced68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:33:54.272 7f577eaced68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:54.272 7f577eaced68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:33:54.272 7f577eaced68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:54.272 7f577eaced68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:54.272 7f577eaced68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:33:54.272 7f577eaced68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:33:58.273 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:34:03.277 7f5781152780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:34:03.277 7f5781152780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:34:03.277 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:03.277 7f5781152780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:34:03.277 7f5781152780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:03.277 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:34:03.278 7f5781152780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:34:03.278 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:03.278 7f5781152780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:34:03.278 7f5781152780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:03.278 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:03.278 7f5781152780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:34:03.278 7f5781152780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:07.281 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:34:12.288 7f5781152780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:34:12.288 7f5781152780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:34:12.288 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:12.288 7f5781152780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:34:12.288 7f5781152780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:12.288 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:34:12.289 7f5781152780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:34:12.289 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:12.289 7f5781152780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:34:12.289 7f5781152780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:12.289 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:12.289 7f5781152780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:34:12.289 7f5781152780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:16.292 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:34:21.298 7f5781152780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:34:21.298 7f5781152780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:34:21.298 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:21.298 7f5781152780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:34:21.298 7f5781152780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:21.298 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:34:21.299 7f5781152780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:34:21.299 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:21.299 7f5781152780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:34:21.299 7f5781152780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:21.299 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:21.299 7f5781152780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:34:21.299 7f5781152780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:25.302 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:34:30.307 7f5781152780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:34:30.307 7f5781152780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:34:30.307 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:30.307 7f5781152780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:34:30.308 7f5781152780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:30.308 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:34:30.308 7f5781152780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:34:30.308 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:30.308 7f5781152780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:34:30.308 7f5781152780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:30.309 7f5781152780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:30.309 7f5781152780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:34:30.309 7f5781152780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:34.309 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:34:39.315 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:34:39.315 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:34:39.315 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:39.315 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:34:39.315 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:39.315 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:34:39.316 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:34:39.316 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:39.316 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:34:39.316 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:39.316 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:39.316 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:34:39.316 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:43.319 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:34:48.322 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:34:48.322 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:34:48.322 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:48.323 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:34:48.323 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:48.323 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:34:48.323 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:34:48.324 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:48.324 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:34:48.324 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:48.324 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:48.324 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:34:48.324 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:52.323 7f577eacec18 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:34:52.539 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51437364, partition_space_available_1k=29653120, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=37372, cpu_share=0.01, mem_res=16060, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34552, cpu_share=0.01, mem_res=14604, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42524, cpu_share=0.01, mem_res=39620, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15310380, free=9414080, buffers=997912, cached=6942748, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.78, five_min_avg=2.83, fifteen_min_avg=3.49, cpu_share=2.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:34:57.330 7f577eacec18 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:34:57.330 7f577eacec18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:34:57.330 7f577eacec18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:57.331 7f577eacec18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:34:57.331 7f577eacec18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:57.331 7f577eacec18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:57.332 7f577eacec18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51437364 partition_space_available_1k = 29653120 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 37372 cpu_share = 0.01 mem_res = 16060 >>contrail-dns : << mem_virt = 34552 cpu_share = 0.01 mem_res = 14604 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42524 cpu_share = 0.01 mem_res = 39620 >> } system_mem_usage = << total = 32665120 used = 15310380 free = 9414080 buffers = 997912 cached = 6942748 node_type = control-node >> system_cpu_usage = << one_min_avg = 1.78 five_min_avg = 2.83 fifteen_min_avg = 3.49 cpu_share = 2.75 node_type = control-node >> >> 06/02/2025 02:34:57.333 7f577eacec18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:34:57.333 7f577eacec18 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:34:57.333 7f577eacec18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:34:57.333 7f577eacec18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:57.334 7f577eacec18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:34:57.334 7f577eacec18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:57.334 7f577eacec18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:34:57.334 7f577eacec18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:34:57.334 7f577eacec18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:01.333 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:35:06.338 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:35:06.339 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:35:06.339 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:06.339 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:35:06.339 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:06.339 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:35:06.339 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:35:06.340 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:06.340 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:35:06.340 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:06.340 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:06.340 7f5781379ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:35:06.340 7f5781379ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:10.341 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:35:15.346 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:35:15.347 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:35:15.347 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:15.347 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:35:15.347 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:15.347 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:35:15.348 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:35:15.348 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:15.348 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:35:15.348 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:15.348 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:15.348 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:35:15.348 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:19.349 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:35:24.354 7f57811c7a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:35:24.354 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:35:24.354 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:24.354 7f57811c7a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:35:24.354 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:24.354 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:35:24.355 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:35:24.355 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:24.355 7f57811c7a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:35:24.355 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:24.355 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:24.355 7f57811c7a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:35:24.356 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:28.356 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:35:33.359 7f57811c7a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:35:33.359 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:35:33.359 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:33.359 7f57811c7a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:35:33.359 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:33.359 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:35:33.360 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:35:33.360 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:33.360 7f57811c7a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:35:33.360 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:33.360 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:33.360 7f57811c7a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:35:33.360 7f57811c7a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:37.362 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:35:42.365 7f577d8a0978 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:35:42.365 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:35:42.365 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:42.365 7f577d8a0978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:35:42.365 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:42.365 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:35:42.366 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:35:42.366 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:42.366 7f577d8a0978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:35:42.366 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:42.366 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:42.366 7f577d8a0978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:35:42.366 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:46.367 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:35:51.370 7f577d8a0978 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:35:51.371 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:35:51.372 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:51.372 7f577d8a0978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:35:51.372 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:35:51.372 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:35:51.373 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:35:51.373 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:51.373 7f577d8a0978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:35:51.373 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:51.373 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:51.373 7f577d8a0978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:35:51.373 7f577d8a0978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:52.547 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51438028, partition_space_available_1k=29652456, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=37356, cpu_share=0.01, mem_res=15908, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34624, cpu_share=0.01, mem_res=14616, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42556, cpu_share=0.01, mem_res=39656, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15317020, free=9406068, buffers=997912, cached=6944120, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=1.98, five_min_avg=2.7, fifteen_min_avg=3.4, cpu_share=2.88, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:35:52.548 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:52.548 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51438028 partition_space_available_1k = 29652456 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 37356 cpu_share = 0.01 mem_res = 15908 >>contrail-dns : << mem_virt = 34624 cpu_share = 0.01 mem_res = 14616 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42556 cpu_share = 0.01 mem_res = 39656 >> } system_mem_usage = << total = 32665120 used = 15317020 free = 9406068 buffers = 997912 cached = 6944120 node_type = control-node >> system_cpu_usage = << one_min_avg = 1.98 five_min_avg = 2.7 fifteen_min_avg = 3.4 cpu_share = 2.88 node_type = control-node >> >> 06/02/2025 02:35:52.548 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:35:55.373 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:36:00.380 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:36:00.380 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:36:00.380 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:00.380 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:36:00.380 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:00.380 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:36:00.381 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:36:00.381 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:00.381 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:36:00.381 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:00.381 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:00.381 7f5781361eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:36:00.381 7f5781361eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:04.381 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:36:08.647 7f577ec1f6d8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:36:09.385 7f577d8b68d0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:36:09.385 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:36:09.385 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:09.385 7f577d8b68d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:36:09.385 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:09.385 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:36:09.386 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:36:09.386 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:09.386 7f577d8b68d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:36:09.386 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:09.386 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:09.387 7f577d8b68d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:36:09.387 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:13.389 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:36:18.391 7f577d8b68d0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:36:18.391 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:36:18.391 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:18.391 7f577d8b68d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:36:18.391 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:18.391 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:36:18.392 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:36:18.392 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:18.392 7f577d8b68d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:36:18.392 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:18.392 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:18.392 7f577d8b68d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:36:18.392 7f577d8b68d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:19.070 7f57811e0ac8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:36:22.393 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:36:27.399 7f57811e0ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:36:27.399 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:36:27.400 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:27.400 7f57811e0ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:36:27.400 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:27.400 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:36:27.400 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:36:27.401 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:27.401 7f57811e0ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:36:27.401 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:27.401 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:27.401 7f57811e0ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:36:27.401 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:31.405 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:36:34.579 7f577d8a02e8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:36:36.409 7f57811e0ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:36:36.409 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:36:36.409 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:36.409 7f57811e0ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:36:36.409 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:36.409 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:36:36.410 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:36:36.410 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:36.410 7f57811e0ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:36:36.410 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:36.410 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:36.410 7f57811e0ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:36:36.410 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:40.410 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:36:45.413 7f57811e0ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:36:45.413 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:36:45.414 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:45.414 7f57811e0ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:36:45.414 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:45.414 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:36:45.415 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:36:45.415 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:45.415 7f57811e0ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:36:45.415 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:45.415 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:45.415 7f57811e0ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:36:45.415 7f57811e0ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:49.415 7f577d8bbeb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:36:52.553 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51448460, partition_space_available_1k=29642024, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=37684, cpu_share=0.01, mem_res=16108, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34688, cpu_share=0.01, mem_res=14620, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42620, cpu_share=0.01, mem_res=39664, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15396340, free=9305684, buffers=997988, cached=6965108, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.66, five_min_avg=3.54, fifteen_min_avg=3.64, cpu_share=5.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:36:54.417 7f577d8bbeb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:36:54.417 7f577d8bbeb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:36:54.417 7f577d8bbeb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:54.417 7f577d8bbeb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:36:54.417 7f577d8bbeb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:54.417 7f577d8bbeb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:54.418 7f577d8bbeb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51448460 partition_space_available_1k = 29642024 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 37684 cpu_share = 0.01 mem_res = 16108 >>contrail-dns : << mem_virt = 34688 cpu_share = 0.01 mem_res = 14620 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42620 cpu_share = 0.01 mem_res = 39664 >> } system_mem_usage = << total = 32665120 used = 15396340 free = 9305684 buffers = 997988 cached = 6965108 node_type = control-node >> system_cpu_usage = << one_min_avg = 5.66 five_min_avg = 3.54 fifteen_min_avg = 3.64 cpu_share = 5.38 node_type = control-node >> >> 06/02/2025 02:36:54.418 7f577d8bbeb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:36:54.418 7f577d8bbeb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:36:54.418 7f577d8bbeb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:36:54.419 7f577d8bbeb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:54.419 7f577d8bbeb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:36:54.419 7f577d8bbeb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:54.419 7f577d8bbeb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:54.419 7f577d8bbeb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:36:54.419 7f577d8bbeb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:36:58.421 7f57811c8438 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:37:03.428 7f57811c8438 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:37:03.428 7f57811c8438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:37:03.428 7f57811c8438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:03.428 7f57811c8438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:37:03.428 7f57811c8438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:03.428 7f57811c8438 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:37:03.429 7f57811c8438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:37:03.429 7f57811c8438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:03.429 7f57811c8438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:37:03.429 7f57811c8438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:03.429 7f57811c8438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:03.429 7f57811c8438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:37:03.429 7f57811c8438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:07.429 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:37:12.434 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:37:12.434 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:37:12.434 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:12.434 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:37:12.435 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:12.435 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:37:12.435 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:37:12.435 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:12.435 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:37:12.435 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:12.435 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:12.436 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:37:12.436 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:16.438 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:37:21.442 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:37:21.442 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:37:21.442 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:21.442 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:37:21.442 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:21.442 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:37:21.443 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:37:21.443 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:21.443 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:37:21.443 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:21.443 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:21.443 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:37:21.443 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:25.445 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:37:30.451 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:37:30.452 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:37:30.452 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:30.452 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:37:30.452 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:30.452 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:37:30.453 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:37:30.453 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:30.453 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:37:30.453 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:30.453 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:30.453 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:37:30.453 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:34.452 7f577cd61828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:37:39.458 7f577cd61828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:37:39.458 7f577cd61828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:37:39.459 7f577cd61828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:39.459 7f577cd61828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:37:39.459 7f577cd61828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:39.459 7f577cd61828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:37:39.459 7f577cd61828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:37:39.459 7f577cd61828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:39.459 7f577cd61828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:37:39.460 7f577cd61828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:39.460 7f577cd61828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:39.460 7f577cd61828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:37:39.460 7f577cd61828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:43.463 7f577cd61828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:37:48.466 7f577cd61828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:37:48.467 7f577cd61828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:37:48.467 7f577cd61828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:48.467 7f577cd61828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:37:48.467 7f577cd61828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:48.467 7f577cd61828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:37:48.468 7f577cd61828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:37:48.468 7f577cd61828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:48.468 7f577cd61828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:37:48.468 7f577cd61828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:48.468 7f577cd61828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:48.468 7f577cd61828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:37:48.468 7f577cd61828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:52.468 7f57811cc828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:37:52.548 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51456036, partition_space_available_1k=29634448, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=37696, cpu_share=0.01, mem_res=15980, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34832, cpu_share=0.01, mem_res=14700, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42736, cpu_share=0.01, mem_res=39664, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15358488, free=9340380, buffers=997988, cached=6968264, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.05, five_min_avg=3.47, fifteen_min_avg=3.61, cpu_share=2.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:37:57.473 7f57811cc828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:37:57.473 7f57811cc828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:37:57.473 7f57811cc828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:57.473 7f57811cc828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:37:57.473 7f57811cc828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:57.474 7f57811cc828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:57.474 7f57811cc828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51456036 partition_space_available_1k = 29634448 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 37696 cpu_share = 0.01 mem_res = 15980 >>contrail-dns : << mem_virt = 34832 cpu_share = 0.01 mem_res = 14700 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42736 cpu_share = 0.01 mem_res = 39664 >> } system_mem_usage = << total = 32665120 used = 15358488 free = 9340380 buffers = 997988 cached = 6968264 node_type = control-node >> system_cpu_usage = << one_min_avg = 4.05 five_min_avg = 3.47 fifteen_min_avg = 3.61 cpu_share = 2.5 node_type = control-node >> >> 06/02/2025 02:37:57.474 7f57811cc828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:37:57.474 7f57811cc828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:37:57.474 7f57811cc828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:37:57.475 7f57811cc828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:57.475 7f57811cc828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:37:57.475 7f57811cc828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:57.475 7f57811cc828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:37:57.476 7f57811cc828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:37:57.476 7f57811cc828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:01.477 7f577d8bbac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:38:06.480 7f577d8bbac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:38:06.481 7f577d8bbac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:38:06.481 7f577d8bbac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:06.481 7f577d8bbac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:38:06.481 7f577d8bbac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:06.481 7f577d8bbac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:38:06.482 7f577d8bbac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:38:06.482 7f577d8bbac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:06.482 7f577d8bbac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:38:06.482 7f577d8bbac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:06.482 7f577d8bbac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:06.482 7f577d8bbac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:38:06.482 7f577d8bbac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:10.484 7f577e308c18 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:38:15.491 7f577e308c18 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:38:15.491 7f577e308c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:38:15.491 7f577e308c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:15.491 7f577e308c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:38:15.491 7f577e308c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:15.491 7f577e308c18 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:38:15.492 7f577e308c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:38:15.492 7f577e308c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:15.492 7f577e308c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:38:15.492 7f577e308c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:15.492 7f577e308c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:15.492 7f577e308c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:38:15.492 7f577e308c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:18.679 7f577e308c18 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:38:19.493 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:38:24.501 7f577ec1f780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:38:24.502 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:38:24.502 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:24.502 7f577ec1f780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:38:24.502 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:24.502 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:38:24.502 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:38:24.502 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:24.503 7f577ec1f780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:38:24.503 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:24.503 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:24.503 7f577ec1f780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:38:24.503 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:28.505 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:38:33.511 7f577ec1f780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:38:33.511 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:38:33.511 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:33.511 7f577ec1f780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:38:33.511 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:33.511 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:38:33.512 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:38:33.512 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:33.512 7f577ec1f780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:38:33.512 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:33.512 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:33.512 7f577ec1f780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:38:33.512 7f577ec1f780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:37.514 7f577cd69c18 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:38:42.521 7f577cd69c18 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:38:42.521 7f577cd69c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:38:42.521 7f577cd69c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:42.521 7f577cd69c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:38:42.521 7f577cd69c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:42.521 7f577cd69c18 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:38:42.522 7f577cd69c18 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:38:42.522 7f577cd69c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:42.522 7f577cd69c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:38:42.522 7f577cd69c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:42.523 7f577cd69c18 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:42.523 7f577cd69c18 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:38:42.523 7f577cd69c18 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:46.521 7f577eacd6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:38:51.524 7f577eacd6d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:38:51.524 7f577eacd6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:38:51.525 7f577eacd6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:51.525 7f577eacd6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:38:51.525 7f577eacd6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:38:51.525 7f577eacd6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:38:51.525 7f577eacd6d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:38:51.525 7f577eacd6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:51.526 7f577eacd6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:38:51.526 7f577eacd6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:51.526 7f577eacd6d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:51.526 7f577eacd6d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:38:51.526 7f577eacd6d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:52.553 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51457156, partition_space_available_1k=29633328, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=37796, cpu_share=0.01, mem_res=15948, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=34904, cpu_share=0.01, mem_res=14708, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42632, cpu_share=0.01, mem_res=39664, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15363132, free=9328668, buffers=997996, cached=6975324, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.04, five_min_avg=3.3, fifteen_min_avg=3.54, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:38:52.553 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:52.553 7f577cd620f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51457156 partition_space_available_1k = 29633328 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 37796 cpu_share = 0.01 mem_res = 15948 >>contrail-dns : << mem_virt = 34904 cpu_share = 0.01 mem_res = 14708 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42632 cpu_share = 0.01 mem_res = 39664 >> } system_mem_usage = << total = 32665120 used = 15363132 free = 9328668 buffers = 997996 cached = 6975324 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.04 five_min_avg = 3.3 fifteen_min_avg = 3.54 cpu_share = 2.62 node_type = control-node >> >> 06/02/2025 02:38:52.553 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:38:55.528 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:39:00.534 7f577cd620f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:39:00.535 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:39:00.535 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:00.535 7f577cd620f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:39:00.535 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:00.535 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:39:00.535 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:39:00.535 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:00.536 7f577cd620f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:39:00.536 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:00.536 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:00.536 7f577cd620f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:39:00.536 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:04.536 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:39:09.542 7f577cd620f0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:39:09.542 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:39:09.542 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:09.543 7f577cd620f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:39:09.543 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:09.543 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:39:09.544 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:39:09.544 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:09.544 7f577cd620f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:39:09.544 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:09.544 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:09.544 7f577cd620f0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:39:09.544 7f577cd620f0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:13.545 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:39:18.551 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:39:18.552 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:39:18.552 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:18.552 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:39:18.552 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:18.552 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:39:18.552 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:39:18.553 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:18.553 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:39:18.553 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:18.553 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:18.553 7f577e2e9630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:39:18.553 7f577e2e9630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:22.554 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:39:27.561 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:39:27.562 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:39:27.562 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:27.562 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:39:27.562 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:27.562 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:39:27.562 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:39:27.563 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:27.563 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:39:27.563 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:27.563 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:27.563 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:39:27.563 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:31.562 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:39:36.566 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:39:36.566 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:39:36.566 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:36.566 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:39:36.566 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:36.566 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:39:36.567 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:39:36.567 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:36.567 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:39:36.567 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:36.567 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:36.567 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:39:36.567 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:40.570 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:39:45.572 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:39:45.573 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:39:45.573 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:45.574 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:39:45.574 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:45.574 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:39:45.575 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:39:45.575 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:45.575 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:39:45.575 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:45.575 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:45.575 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:39:45.575 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:49.573 7f57811e2e10 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:39:52.554 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51457864, partition_space_available_1k=29632620, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=37904, cpu_share=0.01, mem_res=15924, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35044, cpu_share=0.01, mem_res=14788, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.01, mem_res=39668, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15357548, free=9330380, buffers=997996, cached=6979196, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.3, five_min_avg=3.04, fifteen_min_avg=3.44, cpu_share=2.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:39:54.577 7f57811e2e10 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:39:54.577 7f57811e2e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:39:54.577 7f57811e2e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:54.577 7f57811e2e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:39:54.578 7f57811e2e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:54.578 7f57811e2e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:54.578 7f57811e2e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51457864 partition_space_available_1k = 29632620 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 37904 cpu_share = 0.01 mem_res = 15924 >>contrail-dns : << mem_virt = 35044 cpu_share = 0.01 mem_res = 14788 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42692 cpu_share = 0.01 mem_res = 39668 >> } system_mem_usage = << total = 32665120 used = 15357548 free = 9330380 buffers = 997996 cached = 6979196 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.3 five_min_avg = 3.04 fifteen_min_avg = 3.44 cpu_share = 2.5 node_type = control-node >> >> 06/02/2025 02:39:54.578 7f57811e2e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:39:54.578 7f57811e2e10 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:39:54.578 7f57811e2e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:39:54.578 7f57811e2e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:54.578 7f57811e2e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:39:54.578 7f57811e2e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:54.578 7f57811e2e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:54.579 7f57811e2e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:39:54.579 7f57811e2e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:39:58.582 7f5782468630 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:40:03.585 7f5782468630 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:40:03.585 7f5782468630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:40:03.585 7f5782468630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:03.585 7f5782468630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:40:03.585 7f5782468630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:03.585 7f5782468630 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:40:03.586 7f5782468630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:40:03.586 7f5782468630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:03.586 7f5782468630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:40:03.587 7f5782468630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:03.587 7f5782468630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:03.587 7f5782468630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:40:03.587 7f5782468630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:07.585 7f57811c5198 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:40:11.975 7f577d8b2780 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:40:12.588 7f57811c5198 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:40:12.588 7f57811c5198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:40:12.588 7f57811c5198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:12.588 7f57811c5198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:40:12.589 7f57811c5198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:12.589 7f57811c5198 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:40:12.589 7f57811c5198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:40:12.589 7f57811c5198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:12.589 7f57811c5198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:40:12.589 7f57811c5198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:12.589 7f57811c5198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:12.589 7f57811c5198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:40:12.589 7f57811c5198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:16.593 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:40:21.596 7f577d8b2780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:40:21.596 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:40:21.596 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:21.597 7f577d8b2780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:40:21.597 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:21.597 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:40:21.597 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:40:21.597 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:21.597 7f577d8b2780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:40:21.597 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:21.597 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:21.597 7f577d8b2780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:40:21.597 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:22.385 7f577d8b2780 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:40:25.600 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:40:30.606 7f577d8b2780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:40:30.606 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:40:30.607 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:30.607 7f577d8b2780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:40:30.607 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:30.607 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:40:30.608 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:40:30.608 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:30.608 7f577d8b2780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:40:30.608 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:30.608 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:30.608 7f577d8b2780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:40:30.608 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:34.281 7f577d8b2780 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:40:34.608 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:40:39.622 7f577d8b2780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:40:39.623 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:40:39.623 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:39.623 7f577d8b2780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:40:39.623 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:39.623 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:40:39.627 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:40:39.628 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:39.628 7f577d8b2780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:40:39.628 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:39.628 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:39.628 7f577d8b2780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:40:39.629 7f577d8b2780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:43.627 7f57811e0eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:40:48.631 7f57811e0eb8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:40:48.631 7f57811e0eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:40:48.631 7f57811e0eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:48.631 7f57811e0eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:40:48.631 7f57811e0eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:48.631 7f57811e0eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:40:48.632 7f57811e0eb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:40:48.632 7f57811e0eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:48.632 7f57811e0eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:40:48.632 7f57811e0eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:48.632 7f57811e0eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:48.633 7f57811e0eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:40:48.633 7f57811e0eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:52.568 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51466612, partition_space_available_1k=29623872, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=37928, cpu_share=0.01, mem_res=15812, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35136, cpu_share=0.01, mem_res=14820, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42680, cpu_share=0.01, mem_res=39668, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15468404, free=9198760, buffers=998008, cached=6999948, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=8.82, five_min_avg=4.53, fifteen_min_avg=3.91, cpu_share=5.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:40:52.569 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:52.569 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51466612 partition_space_available_1k = 29623872 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 37928 cpu_share = 0.01 mem_res = 15812 >>contrail-dns : << mem_virt = 35136 cpu_share = 0.01 mem_res = 14820 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42680 cpu_share = 0.01 mem_res = 39668 >> } system_mem_usage = << total = 32665120 used = 15468404 free = 9198760 buffers = 998008 cached = 6999948 node_type = control-node >> system_cpu_usage = << one_min_avg = 8.82 five_min_avg = 4.53 fifteen_min_avg = 3.91 cpu_share = 5.25 node_type = control-node >> >> 06/02/2025 02:40:52.569 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:52.632 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:40:57.638 7f577ec35828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:40:57.638 7f577ec35828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:40:57.639 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:57.639 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:40:57.639 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:40:57.639 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:40:57.639 7f577ec35828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:40:57.639 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:57.639 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:40:57.639 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:57.640 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:40:57.640 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:40:57.640 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:01.641 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:41:06.643 7f577ec35828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:41:06.643 7f577ec35828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:41:06.643 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:06.644 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:41:06.644 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:06.644 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:41:06.644 7f577ec35828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:41:06.644 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:06.645 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:41:06.645 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:06.645 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:06.645 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:41:06.645 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:10.647 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:41:15.653 7f577ec35828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:41:15.654 7f577ec35828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:41:15.654 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:15.654 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:41:15.654 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:15.654 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:41:15.654 7f577ec35828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:41:15.654 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:15.654 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:41:15.654 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:15.655 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:15.655 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:41:15.655 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:19.656 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:41:24.661 7f577ec35828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:41:24.661 7f577ec35828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:41:24.661 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:24.661 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:41:24.661 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:24.661 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:41:24.662 7f577ec35828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:41:24.662 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:24.662 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:41:24.662 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:24.662 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:24.662 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:41:24.662 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:28.665 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:41:33.672 7f577ec35828 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:41:33.672 7f577ec35828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:41:33.672 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:33.672 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:41:33.673 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:33.673 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:41:33.673 7f577ec35828 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:41:33.673 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:33.673 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:41:33.674 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:33.674 7f577ec35828 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:33.674 7f577ec35828 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:41:33.674 7f577ec35828 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:37.673 7f577eacf630 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:41:42.679 7f577eacf630 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:41:42.680 7f577eacf630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:41:42.680 7f577eacf630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:42.680 7f577eacf630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:41:42.680 7f577eacf630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:42.680 7f577eacf630 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:41:42.681 7f577eacf630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:41:42.681 7f577eacf630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:42.681 7f577eacf630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:41:42.681 7f577eacf630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:42.682 7f577eacf630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:42.682 7f577eacf630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:41:42.682 7f577eacf630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:46.680 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:41:51.682 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:41:51.682 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:41:51.682 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:51.683 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:41:51.683 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:41:51.683 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:41:51.683 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:41:51.684 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:51.684 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:41:51.684 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:51.684 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:51.684 7f57875f0a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:41:51.684 7f57875f0a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:52.568 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51470436, partition_space_available_1k=29620048, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=38084, cpu_share=0.01, mem_res=15836, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35200, cpu_share=0.01, mem_res=14820, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42708, cpu_share=0.01, mem_res=39668, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15421120, free=9243100, buffers=998008, cached=7002892, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.29, five_min_avg=4.03, fifteen_min_avg=3.78, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:41:52.568 7f577d741f60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:52.569 7f577d741f60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51470436 partition_space_available_1k = 29620048 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 38084 cpu_share = 0.01 mem_res = 15836 >>contrail-dns : << mem_virt = 35200 cpu_share = 0.01 mem_res = 14820 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42708 cpu_share = 0.01 mem_res = 39668 >> } system_mem_usage = << total = 32665120 used = 15421120 free = 9243100 buffers = 998008 cached = 7002892 node_type = control-node >> system_cpu_usage = << one_min_avg = 4.29 five_min_avg = 4.03 fifteen_min_avg = 3.78 cpu_share = 2.62 node_type = control-node >> >> 06/02/2025 02:41:52.569 7f577d741f60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:41:55.685 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:42:00.692 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:42:00.692 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:42:00.692 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:00.692 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:42:00.692 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:00.692 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:42:00.693 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:42:00.693 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:00.693 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:42:00.693 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:00.693 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:00.693 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:42:00.693 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:04.696 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:42:09.700 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:42:09.700 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:42:09.700 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:09.700 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:42:09.700 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:09.701 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:42:09.701 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:42:09.702 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:09.702 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:42:09.702 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:09.702 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:09.702 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:42:09.702 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:13.704 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:42:18.709 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:42:18.709 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:42:18.709 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:18.709 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:42:18.709 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:18.709 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:42:18.709 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:42:18.710 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:18.710 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:42:18.710 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:18.710 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:18.710 7f577e2e5e10 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:42:18.710 7f577e2e5e10 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:22.710 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:42:27.716 7f577d8b2048 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:42:27.716 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:42:27.716 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:27.716 7f577d8b2048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:42:27.717 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:27.717 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:42:27.717 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:42:27.717 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:27.717 7f577d8b2048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:42:27.717 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:27.717 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:27.717 7f577d8b2048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:42:27.717 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:31.721 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:42:36.727 7f577d8b2048 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:42:36.728 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:42:36.728 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:36.728 7f577d8b2048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:42:36.728 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:36.728 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:42:36.729 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:42:36.729 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:36.729 7f577d8b2048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:42:36.729 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:36.729 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:36.729 7f577d8b2048 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:42:36.729 7f577d8b2048 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:40.731 7f577eaceb70 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:42:45.734 7f577eaceb70 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:42:45.735 7f577eaceb70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:42:45.735 7f577eaceb70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:45.735 7f577eaceb70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:42:45.735 7f577eaceb70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:45.735 7f577eaceb70 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:42:45.736 7f577eaceb70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:42:45.736 7f577eaceb70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:45.737 7f577eaceb70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:42:45.737 7f577eaceb70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:45.737 7f577eaceb70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:45.737 7f577eaceb70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:42:45.737 7f577eaceb70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:49.735 7f577d8b6630 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:42:52.569 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51476600, partition_space_available_1k=29613884, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=38352, cpu_share=0.01, mem_res=15968, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35260, cpu_share=0.01, mem_res=14820, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42772, cpu_share=0.01, mem_res=39676, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15419928, free=9241864, buffers=998008, cached=7005320, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.48, five_min_avg=3.86, fifteen_min_avg=3.73, cpu_share=2.62, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:42:54.739 7f577d8b6630 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:42:54.740 7f577d8b6630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:42:54.740 7f577d8b6630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:54.740 7f577d8b6630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:42:54.740 7f577d8b6630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:54.740 7f577d8b6630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:54.740 7f577d8b6630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51476600 partition_space_available_1k = 29613884 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 38352 cpu_share = 0.01 mem_res = 15968 >>contrail-dns : << mem_virt = 35260 cpu_share = 0.01 mem_res = 14820 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42772 cpu_share = 0.01 mem_res = 39676 >> } system_mem_usage = << total = 32665120 used = 15419928 free = 9241864 buffers = 998008 cached = 7005320 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.48 five_min_avg = 3.86 fifteen_min_avg = 3.73 cpu_share = 2.62 node_type = control-node >> >> 06/02/2025 02:42:54.740 7f577d8b6630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:42:54.740 7f577d8b6630 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:42:54.741 7f577d8b6630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:42:54.741 7f577d8b6630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:54.741 7f577d8b6630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:42:54.741 7f577d8b6630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:54.741 7f577d8b6630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:54.741 7f577d8b6630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:42:54.741 7f577d8b6630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:42:58.743 7f577d8a08d0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:43:03.749 7f577d8a08d0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:43:03.749 7f577d8a08d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:43:03.749 7f577d8a08d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:03.749 7f577d8a08d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:43:03.749 7f577d8a08d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:03.749 7f577d8a08d0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:43:03.750 7f577d8a08d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:43:03.750 7f577d8a08d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:03.750 7f577d8a08d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:43:03.750 7f577d8a08d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:03.750 7f577d8a08d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:03.750 7f577d8a08d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:43:03.750 7f577d8a08d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:07.750 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:43:10.679 7f5782462ac8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:43:12.753 7f5780cb3390 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:43:12.754 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:43:12.754 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:12.754 7f5780cb3390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:43:12.754 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:12.754 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:43:12.755 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:43:12.755 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:12.755 7f5780cb3390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:43:12.755 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:12.755 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:12.755 7f5780cb3390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:43:12.755 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:16.757 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:43:21.764 7f5780cb3390 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:43:21.764 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:43:21.765 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:21.765 7f5780cb3390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:43:21.765 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:21.766 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:43:21.766 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:43:21.766 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:21.767 7f5780cb3390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:43:21.767 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:21.767 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:21.767 7f5780cb3390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:43:21.767 7f5780cb3390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:25.766 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:43:30.773 7f57811b1cc0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:43:30.773 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:43:30.773 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:30.773 7f57811b1cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:43:30.773 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:30.773 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:43:30.774 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:43:30.774 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:30.774 7f57811b1cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:43:30.774 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:30.774 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:30.774 7f57811b1cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:43:30.774 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:34.777 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:43:39.778 7f57811b1cc0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:43:39.778 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:43:39.779 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:39.779 7f57811b1cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:43:39.779 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:39.779 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:43:39.779 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:43:39.779 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:39.779 7f57811b1cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:43:39.779 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:39.780 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:39.780 7f57811b1cc0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:43:39.780 7f57811b1cc0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:43.781 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:43:48.784 7f577e2f06d8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:43:48.784 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:43:48.784 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:48.784 7f577e2f06d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:43:48.784 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:48.785 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:43:48.786 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:43:48.786 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:48.786 7f577e2f06d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:43:48.786 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:48.786 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:48.786 7f577e2f06d8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:43:48.786 7f577e2f06d8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:52.558 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51477980, partition_space_available_1k=29612504, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=38464, cpu_share=0.01, mem_res=15948, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35340, cpu_share=0.01, mem_res=14832, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42760, cpu_share=0.01, mem_res=39676, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15417532, free=9236824, buffers=998008, cached=7012756, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.04, five_min_avg=3.68, fifteen_min_avg=3.68, cpu_share=2.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:43:52.558 7f577ead3240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:52.559 7f577ead3240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51477980 partition_space_available_1k = 29612504 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 38464 cpu_share = 0.01 mem_res = 15948 >>contrail-dns : << mem_virt = 35340 cpu_share = 0.01 mem_res = 14832 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42760 cpu_share = 0.01 mem_res = 39676 >> } system_mem_usage = << total = 32665120 used = 15417532 free = 9236824 buffers = 998008 cached = 7012756 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.04 five_min_avg = 3.68 fifteen_min_avg = 3.68 cpu_share = 2.75 node_type = control-node >> >> 06/02/2025 02:43:52.559 7f577ead3240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:52.784 7f577ead3630 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:43:57.789 7f577ead3630 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:43:57.789 7f577ead3630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:43:57.789 7f577ead3630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:57.789 7f577ead3630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:43:57.790 7f577ead3630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:43:57.790 7f577ead3630 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:43:57.790 7f577ead3630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:43:57.790 7f577ead3630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:57.790 7f577ead3630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:43:57.790 7f577ead3630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:57.790 7f577ead3630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:43:57.790 7f577ead3630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:43:57.790 7f577ead3630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:01.793 7f577ead3630 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:44:06.799 7f577ead3630 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:44:06.799 7f577ead3630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:44:06.800 7f577ead3630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:06.800 7f577ead3630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:44:06.800 7f577ead3630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:06.800 7f577ead3630 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:44:06.800 7f577ead3630 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:44:06.800 7f577ead3630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:06.801 7f577ead3630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:44:06.801 7f577ead3630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:06.801 7f577ead3630 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:06.801 7f577ead3630 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:44:06.801 7f577ead3630 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:10.801 7f577e308588 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:44:15.806 7f577e308588 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:44:15.807 7f577e308588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:44:15.807 7f577e308588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:15.807 7f577e308588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:44:15.807 7f577e308588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:15.807 7f577e308588 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:44:15.808 7f577e308588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:44:15.808 7f577e308588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:15.808 7f577e308588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:44:15.808 7f577e308588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:15.808 7f577e308588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:15.808 7f577e308588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:44:15.808 7f577e308588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:19.810 7f577e308588 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:44:24.814 7f577e308588 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:44:24.814 7f577e308588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:44:24.814 7f577e308588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:24.814 7f577e308588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:44:24.814 7f577e308588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:24.814 7f577e308588 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:44:24.815 7f577e308588 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:44:24.815 7f577e308588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:24.815 7f577e308588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:44:24.815 7f577e308588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:24.816 7f577e308588 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:24.816 7f577e308588 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:44:24.816 7f577e308588 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:28.818 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:44:33.821 7f577ead3d68 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:44:33.821 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:44:33.821 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:33.821 7f577ead3d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:44:33.821 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:33.821 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:44:33.822 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:44:33.822 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:33.822 7f577ead3d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:44:33.822 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:33.822 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:33.822 7f577ead3d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:44:33.822 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:37.825 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:44:42.831 7f577ead3d68 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:44:42.831 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:44:42.831 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:42.831 7f577ead3d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:44:42.831 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:42.831 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:44:42.832 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:44:42.832 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:42.832 7f577ead3d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:44:42.832 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:42.832 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:42.832 7f577ead3d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:44:42.832 7f577ead3d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:46.832 7f577ead0390 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:44:51.836 7f577ead0390 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:44:51.836 7f577ead0390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:44:51.836 7f577ead0390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:51.836 7f577ead0390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:44:51.836 7f577ead0390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:44:51.836 7f577ead0390 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:44:51.837 7f577ead0390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:44:51.837 7f577ead0390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:51.837 7f577ead0390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:44:51.837 7f577ead0390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:51.837 7f577ead0390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:51.837 7f577ead0390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:44:51.837 7f577ead0390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:52.567 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51478624, partition_space_available_1k=29611860, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=38608, cpu_share=0.01, mem_res=15956, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35488, cpu_share=0.01, mem_res=14920, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42788, cpu_share=0.01, mem_res=39676, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15411332, free=9241484, buffers=998008, cached=7014296, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.51, five_min_avg=3.39, fifteen_min_avg=3.58, cpu_share=2.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:44:52.567 7f577eacff60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:52.567 7f577eacff60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51478624 partition_space_available_1k = 29611860 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 38608 cpu_share = 0.01 mem_res = 15956 >>contrail-dns : << mem_virt = 35488 cpu_share = 0.01 mem_res = 14920 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42788 cpu_share = 0.01 mem_res = 39676 >> } system_mem_usage = << total = 32665120 used = 15411332 free = 9241484 buffers = 998008 cached = 7014296 node_type = control-node >> system_cpu_usage = << one_min_avg = 2.51 five_min_avg = 3.39 fifteen_min_avg = 3.58 cpu_share = 2.75 node_type = control-node >> >> 06/02/2025 02:44:52.567 7f577eacff60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:44:55.838 7f577ead0198 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:00.843 7f577ead0198 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:45:00.843 7f577ead0198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:45:00.843 7f577ead0198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:00.843 7f577ead0198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:45:00.844 7f577ead0198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:00.844 7f577ead0198 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:45:00.844 7f577ead0198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:45:00.844 7f577ead0198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:00.845 7f577ead0198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:45:00.845 7f577ead0198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:00.845 7f577ead0198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:00.845 7f577ead0198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:45:00.845 7f577ead0198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:04.845 7f577ead0198 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:09.848 7f577ead0198 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:45:09.849 7f577ead0198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:45:09.849 7f577ead0198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:09.849 7f577ead0198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:45:09.849 7f577ead0198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:09.849 7f577ead0198 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:45:09.850 7f577ead0198 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:45:09.850 7f577ead0198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:09.850 7f577ead0198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:45:09.850 7f577ead0198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:09.850 7f577ead0198 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:09.850 7f577ead0198 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:45:09.850 7f577ead0198 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:13.852 7f57811ae240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:18.857 7f57811ae240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:45:18.857 7f57811ae240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:45:18.857 7f57811ae240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:18.857 7f57811ae240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:45:18.857 7f57811ae240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:18.857 7f57811ae240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:45:18.858 7f57811ae240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:45:18.858 7f57811ae240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:18.858 7f57811ae240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:45:18.858 7f57811ae240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:18.858 7f57811ae240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:18.859 7f57811ae240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:45:18.859 7f57811ae240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:22.860 7f57811ae240 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:27.866 7f57811ae240 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:45:27.867 7f57811ae240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:45:27.867 7f57811ae240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:27.867 7f57811ae240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:45:27.867 7f57811ae240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:27.867 7f57811ae240 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:45:27.867 7f57811ae240 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:45:27.867 7f57811ae240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:27.867 7f57811ae240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:45:27.867 7f57811ae240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:27.867 7f57811ae240 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:27.868 7f57811ae240 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:45:27.868 7f57811ae240 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:31.871 7f577ead3390 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:36.877 7f577ead3390 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:45:36.877 7f577ead3390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:45:36.877 7f577ead3390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:36.877 7f577ead3390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:45:36.878 7f577ead3390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:36.878 7f577ead3390 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:45:36.878 7f577ead3390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:45:36.878 7f577ead3390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:36.878 7f577ead3390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:45:36.878 7f577ead3390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:36.878 7f577ead3390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:36.878 7f577ead3390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:45:36.878 7f577ead3390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:40.878 7f57811cc978 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:45.882 7f57811cc978 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:45:45.882 7f57811cc978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:45:45.882 7f57811cc978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:45.882 7f57811cc978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:45:45.883 7f57811cc978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:45.883 7f57811cc978 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:45:45.883 7f57811cc978 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:45:45.884 7f57811cc978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:45.884 7f57811cc978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:45:45.884 7f57811cc978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:45.884 7f57811cc978 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:45.884 7f57811cc978 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:45:45.884 7f57811cc978 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:49.797 7f577eaf1390 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:45:49.882 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:52.570 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51485264, partition_space_available_1k=29605220, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=38784, cpu_share=0.01, mem_res=16000, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35576, cpu_share=0.01, mem_res=14944, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42840, cpu_share=0.01, mem_res=39676, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15520912, free=9110012, buffers=998008, cached=7036188, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=7.08, five_min_avg=4.57, fifteen_min_avg=3.97, cpu_share=5.25, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:45:54.884 7f577eaf1390 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:45:54.885 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:45:54.885 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:54.885 7f577eaf1390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:45:54.885 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:54.885 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:54.885 7f577eaf1390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51485264 partition_space_available_1k = 29605220 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 38784 cpu_share = 0.01 mem_res = 16000 >>contrail-dns : << mem_virt = 35576 cpu_share = 0.01 mem_res = 14944 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42840 cpu_share = 0.01 mem_res = 39676 >> } system_mem_usage = << total = 32665120 used = 15520912 free = 9110012 buffers = 998008 cached = 7036188 node_type = control-node >> system_cpu_usage = << one_min_avg = 7.08 five_min_avg = 4.57 fifteen_min_avg = 3.97 cpu_share = 5.25 node_type = control-node >> >> 06/02/2025 02:45:54.885 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:45:54.885 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:45:54.886 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:45:54.886 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:54.886 7f577eaf1390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:45:54.886 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:54.886 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:54.886 7f577eaf1390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:45:54.886 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:45:58.888 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:45:59.880 7f5781361eb8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:46:03.893 7f577eaf1390 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:46:03.893 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:46:03.893 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:03.893 7f577eaf1390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:46:03.893 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:03.893 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:46:03.894 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:46:03.894 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:03.894 7f577eaf1390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:46:03.894 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:03.894 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:03.894 7f577eaf1390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:46:03.894 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:07.897 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:46:10.797 7f5780ccea20 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:46:12.901 7f577eaf1390 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:46:12.901 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:46:12.902 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:12.902 7f577eaf1390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:46:12.902 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:12.902 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:46:12.902 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:46:12.903 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:12.903 7f577eaf1390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:46:12.903 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:12.903 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:12.903 7f577eaf1390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:46:12.903 7f577eaf1390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:16.905 7f577ead3780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:46:21.913 7f577ead3780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:46:21.913 7f577ead3780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:46:21.914 7f577ead3780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:21.914 7f577ead3780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:46:21.914 7f577ead3780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:21.914 7f577ead3780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:46:21.915 7f577ead3780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:46:21.915 7f577ead3780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:21.915 7f577ead3780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:46:21.915 7f577ead3780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:21.915 7f577ead3780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:21.915 7f577ead3780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:46:21.916 7f577ead3780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:25.914 7f577eaf1438 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:46:30.921 7f577eaf1438 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:46:30.921 7f577eaf1438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:46:30.921 7f577eaf1438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:30.921 7f577eaf1438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:46:30.921 7f577eaf1438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:30.921 7f577eaf1438 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:46:30.921 7f577eaf1438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:46:30.922 7f577eaf1438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:30.922 7f577eaf1438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:46:30.922 7f577eaf1438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:30.922 7f577eaf1438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:30.922 7f577eaf1438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:46:30.922 7f577eaf1438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:34.925 7f57811c5780 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:46:39.929 7f57811c5780 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:46:39.929 7f57811c5780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:46:39.929 7f57811c5780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:39.929 7f57811c5780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:46:39.929 7f57811c5780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:39.929 7f57811c5780 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:46:39.930 7f57811c5780 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:46:39.930 7f57811c5780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:39.930 7f57811c5780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:46:39.930 7f57811c5780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:39.930 7f57811c5780 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:39.930 7f57811c5780 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:46:39.930 7f57811c5780 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:43.933 7f577eafff60 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:46:48.935 7f577eafff60 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:46:48.935 7f577eafff60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:46:48.936 7f577eafff60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:48.936 7f577eafff60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:46:48.936 7f577eafff60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:48.936 7f577eafff60 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:46:48.937 7f577eafff60 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:46:48.937 7f577eafff60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:48.937 7f577eafff60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:46:48.937 7f577eafff60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:48.937 7f577eafff60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:48.937 7f577eafff60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:46:48.937 7f577eafff60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:52.586 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51490936, partition_space_available_1k=29599548, percentage_partition_space_used=63, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=38992, cpu_share=0.01, mem_res=16072, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35636, cpu_share=0.01, mem_res=14944, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42832, cpu_share=0.01, mem_res=39676, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15432860, free=9196140, buffers=998012, cached=7038108, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.95, five_min_avg=4.14, fifteen_min_avg=3.86, cpu_share=3.12, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:46:52.586 7f577eafff60 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:52.586 7f577eafff60 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51490936 partition_space_available_1k = 29599548 percentage_partition_space_used = 63 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 38992 cpu_share = 0.01 mem_res = 16072 >>contrail-dns : << mem_virt = 35636 cpu_share = 0.01 mem_res = 14944 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42832 cpu_share = 0.01 mem_res = 39676 >> } system_mem_usage = << total = 32665120 used = 15432860 free = 9196140 buffers = 998012 cached = 7038108 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.95 five_min_avg = 4.14 fifteen_min_avg = 3.86 cpu_share = 3.12 node_type = control-node >> >> 06/02/2025 02:46:52.586 7f577eafff60 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:52.936 7f5781152b70 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:46:57.941 7f5781152b70 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:46:57.941 7f5781152b70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:46:57.941 7f5781152b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:57.942 7f5781152b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:46:57.942 7f5781152b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:46:57.942 7f5781152b70 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:46:57.942 7f5781152b70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:46:57.942 7f5781152b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:57.942 7f5781152b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:46:57.942 7f5781152b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:57.942 7f5781152b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:46:57.942 7f5781152b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:46:57.942 7f5781152b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:01.946 7f5781152b70 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:47:06.951 7f5781152b70 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:47:06.951 7f5781152b70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:47:06.952 7f5781152b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:06.952 7f5781152b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:47:06.952 7f5781152b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:06.952 7f5781152b70 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:47:06.953 7f5781152b70 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:47:06.953 7f5781152b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:06.953 7f5781152b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:47:06.953 7f5781152b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:06.953 7f5781152b70 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:06.953 7f5781152b70 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:47:06.953 7f5781152b70 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:10.951 7f577eacf390 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:47:15.957 7f577eacf390 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:47:15.957 7f577eacf390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:47:15.958 7f577eacf390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:15.958 7f577eacf390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:47:15.958 7f577eacf390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:15.958 7f577eacf390 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:47:15.959 7f577eacf390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:47:15.959 7f577eacf390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:15.959 7f577eacf390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:47:15.959 7f577eacf390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:15.959 7f577eacf390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:15.959 7f577eacf390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:47:15.959 7f577eacf390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:19.961 7f577eacf390 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:47:21.266 7f577d8b2ac8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:47:24.967 7f577eacf390 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:47:24.967 7f577eacf390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:47:24.967 7f577eacf390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:24.967 7f577eacf390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:47:24.967 7f577eacf390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:24.967 7f577eacf390 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:47:24.968 7f577eacf390 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:47:24.968 7f577eacf390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:24.968 7f577eacf390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:47:24.968 7f577eacf390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:24.968 7f577eacf390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:24.968 7f577eacf390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:47:24.968 7f577eacf390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:28.971 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:47:33.977 7f5780ccea20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:47:33.977 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:47:33.977 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:33.977 7f5780ccea20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:47:33.977 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:33.977 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:47:33.978 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:47:33.978 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:33.978 7f5780ccea20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:47:33.978 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:33.978 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:33.978 7f5780ccea20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:47:33.978 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:37.981 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:47:42.985 7f5780ccea20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:47:42.985 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:47:42.985 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:42.985 7f5780ccea20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:47:42.985 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:42.985 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:47:42.986 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:47:42.986 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:42.986 7f5780ccea20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:47:42.986 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:42.986 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:42.986 7f5780ccea20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:47:42.986 7f5780ccea20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:46.986 7f577e308ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:47:51.990 7f577e308ac8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:47:51.990 7f577e308ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:47:51.990 7f577e308ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:51.990 7f577e308ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:47:51.990 7f577e308ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:47:51.990 7f577e308ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:47:51.991 7f577e308ac8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:47:51.991 7f577e308ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:51.991 7f577e308ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:47:51.991 7f577e308ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:51.991 7f577e308ac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:51.991 7f577e308ac8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:47:51.991 7f577e308ac8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:52.587 7f57813cbe48 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-juju-os-542-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=51497492, partition_space_available_1k=29592992, percentage_partition_space_used=64, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=39152, cpu_share=0.01, mem_res=16096, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=35700, cpu_share=0.01, mem_res=14944, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42692, cpu_share=0.0, mem_res=37748, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42856, cpu_share=0.01, mem_res=39676, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665120, used=15431228, free=9190048, buffers=998012, cached=7045832, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.8, five_min_avg=4.06, fifteen_min_avg=3.85, cpu_share=3.0, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/02/2025 02:47:52.587 7f577eacf390 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:52.587 7f577eacf390 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 51497492 partition_space_available_1k = 29592992 percentage_partition_space_used = 64 >> } process_mem_cpu_usage = { contrail-control : << mem_virt = 39152 cpu_share = 0.01 mem_res = 16096 >>contrail-dns : << mem_virt = 35700 cpu_share = 0.01 mem_res = 14944 >>contrail-named : << mem_virt = 42692 cpu_share = 0.0 mem_res = 37748 >>contrail-control-nodemgr : << mem_virt = 42856 cpu_share = 0.01 mem_res = 39676 >> } system_mem_usage = << total = 32665120 used = 15431228 free = 9190048 buffers = 998012 cached = 7045832 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.8 five_min_avg = 4.06 fifteen_min_avg = 3.85 cpu_share = 3.0 node_type = control-node >> >> 06/02/2025 02:47:52.587 7f577eacf390 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:47:55.990 7f577cd62438 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:48:00.996 7f577cd62438 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:48:00.996 7f577cd62438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:48:00.996 7f577cd62438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:00.996 7f577cd62438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:48:00.997 7f577cd62438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:00.997 7f577cd62438 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:48:00.997 7f577cd62438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:48:00.997 7f577cd62438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:00.997 7f577cd62438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:48:00.997 7f577cd62438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:00.998 7f577cd62438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:00.998 7f577cd62438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:48:00.998 7f577cd62438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:05.000 7f577cd62438 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:48:10.006 7f577cd62438 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:48:10.006 7f577cd62438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:48:10.007 7f577cd62438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:10.007 7f577cd62438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:48:10.007 7f577cd62438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:10.007 7f577cd62438 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:48:10.007 7f577cd62438 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:48:10.007 7f577cd62438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:10.008 7f577cd62438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:48:10.008 7f577cd62438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:10.008 7f577cd62438 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:10.008 7f577cd62438 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:48:10.008 7f577cd62438 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:14.009 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:48:19.013 7f577ead0d68 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:48:19.014 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:48:19.014 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:19.014 7f577ead0d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:48:19.014 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:19.014 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:48:19.015 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:48:19.015 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:19.015 7f577ead0d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:48:19.015 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:19.015 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:19.015 7f577ead0d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:48:19.016 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:23.017 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/02/2025 02:48:23.097 7f57811ccb70 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/02/2025 02:48:28.021 7f577ead0d68 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/02/2025 02:48:28.021 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/02/2025 02:48:28.021 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:28.021 7f577ead0d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/02/2025 02:48:28.021 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/02/2025 02:48:28.021 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/02/2025 02:48:28.022 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/02/2025 02:48:28.022 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:28.022 7f577ead0d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-542-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.129:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/02/2025 02:48:28.022 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:28.022 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/02/2025 02:48:28.022 7f577ead0d68 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-542-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1748830003869486 collector_name = collector_ip = 10.0.0.129:8086 collector_list = [ 10.0.0.129:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 06/02/2025 02:48:28.022 7f577ead0d68 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]