06/09/2025 02:14:53.256 7f5202860388 [contrail-control-nodemgr] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 06/09/2025 02:14:53.256 7f5202860388 [contrail-control-nodemgr] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 06/09/2025 02:14:53.256 7f5202860388 [contrail-control-nodemgr] [INFO]: SANDESH: Flow Logging: [None] -> [False] 06/09/2025 02:14:53.265 7f51fe1080f0 [contrail-control-nodemgr] [ERROR]: Cannot write http_port 8101 to /tmp/contrail-control-nodemgr.0.http_port 06/09/2025 02:14:53.265 7f51fe1080f0 [contrail-control-nodemgr] [ERROR]: Starting Introspect on HTTP Port 8101 06/09/2025 02:14:53.266 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvStart] in state[none] 06/09/2025 02:14:53.266 7f51fe10ca20 [contrail-control-nodemgr] [DEBUG]: Add uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/09/2025 02:14:53.267 7f51fe10ca20 [contrail-control-nodemgr] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/09/2025 02:14:53.267 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 06/09/2025 02:14:53.267 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:14:53.267 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/09/2025 02:14:53.267 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:14:53.268 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:14:53.268 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 06/09/2025 02:14:53.268 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:14:53.409 7f5202860388 [contrail-control-nodemgr] [DEBUG]: DBG: get_current_processes: '[{'name': 'contrail-control', 'group': 'contrail-control', 'pid': 5232304725075150099353179023475812663811212989876496510037498098698037192366, 'start': '1749435291000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-dns', 'group': 'contrail-dns', 'pid': 12104563503617042570649803573967643171339899187922275648673961319953682624211, 'start': '1749435291000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-named', 'group': 'contrail-named', 'pid': 101984743269103665171795963241352538301744636956012754296529398347435286531568, 'start': '1749435292000000', 'statename': 'PROCESS_STATE_RUNNING'}, {'name': 'contrail-control-nodemgr', 'group': 'contrail-control-nodemgr', 'pid': 89075482293218902491391832112133971516368460518001021309751105193831964315762, 'start': '1749435291000000', 'statename': 'PROCESS_STATE_RUNNING'}]' 06/09/2025 02:14:53.557 7f5202860388 [contrail-control-nodemgr] [INFO]: send_nodemgr_process_status: Sending UVE:NodeStatusUVE(_context='', _scope='', _module='contrail-control-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3422-1', _node_type='Control', _instance_id='0', _timestamp=1749435293557669, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3422-1', deleted=None, status=None, process_status=[ProcessStatus(module_id='contrail-control-nodemgr', instance_id='0', state='Non-Functional', connection_infos=dict_values([ConnectionInfo(type='Collector', name='Collector', server_addrs=[''], status='Down', description='none to Idle on EvStart')]), flag_infos=None, description='Collector:Collector[none to Idle on EvStart] connection down')], process_info=None, disk_usage_info=None, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter'), _seqnum=0) 06/09/2025 02:14:53.558 7f5202860388 [contrail-control-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-control-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3422-1', _node_type='Control', _instance_id='0', _timestamp=1749435293558539, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3422-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='contrail-control', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1749435291000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-dns', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1749435291000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-named', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1749435292000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-control-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1749435291000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-09 00:44:55.879161", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9411-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter'), _seqnum=0) 06/09/2025 02:14:53.562 7f51ffff6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:14:53.564 7f51ffff6eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-09 00:44:55.879161", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9411-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} system_cpu_info = << num_socket = 1 num_cpu = 8 num_core_per_socket = 8 num_thread_per_core = 1 >> running_package_version = nightly-nightly.el7 installed_package_version = nightly-nightly.el7 >> 06/09/2025 02:14:53.564 7f51ffff6eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:14:53.564 7f51ffff6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:14:53.564 7f51ffff6eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/09/2025 02:14:53.564 7f51ffff6eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:14:53.564 7f51ffff6eb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:14:53.565 7f51ffff6eb8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 deleted = False process_info = [ << process_name = contrail-control process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1749435291000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-dns process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1749435291000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-named process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1749435292000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-control-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1749435291000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-09 00:44:55.879161", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9411-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 06/09/2025 02:14:53.565 7f51ffff6eb8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:14:57.267 7f51fffe92e8 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:14:57.268 7f51fffe92e8 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:14:57.268 7f51fffe92e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:14:57.268 7f51fffe92e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:14:57.268 7f51fffe92e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:14:57.268 7f51fffe92e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:14:57.269 7f51fffe92e8 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:14:57.269 7f51fffe92e8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:14:57.269 7f51fffe92e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:14:57.269 7f51fffe92e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:14:57.269 7f51fffe92e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:14:57.269 7f51fffe92e8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:14:57.269 7f51fffe92e8 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:14:57.269 7f51fffe92e8 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:01.192 7f5202860388 [contrail-control-nodemgr] [INFO]: send_process_state_db: Sending UVE: NodeStatusUVE(_context='', _scope='', _module='contrail-control-nodemgr', _source='cn-jenkins-deploy-platform-ansible-os-3422-1', _node_type='Control', _instance_id='0', _timestamp=1749435301192670, _versionsig=3672106175, _type=6, _hints=1, _client_context='', _client=None, _role=0, _logger=None, _level=2147483647, _category='', _send_queue_enabled=True, _http_server=None, _connect_to_collector=True, _disable_sending_object_logs=False, _disable_sending_all_messages=False, _more=False, data=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3422-1', deleted=False, status=None, process_status=None, process_info=[ProcessInfo(process_name='contrail-control', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1749435291000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-dns', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1749435291000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-named', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1749435292000000', last_stop_time='', last_exit_time='', core_file_list=[]), ProcessInfo(process_name='contrail-control-nodemgr', process_state='PROCESS_STATE_RUNNING', start_count=1, stop_count=0, exit_count=0, last_start_time='1749435291000000', last_stop_time='', last_exit_time='', core_file_list=[])], disk_usage_info=None, description=None, all_core_file_list=None, build_info='{"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-09 00:44:55.879161", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9411-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]}', process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, system_cpu_usage=None, running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter'), _seqnum=0) 06/09/2025 02:15:01.193 7f5202860388 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3422-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=14874584, partition_space_available_1k=66215900, percentage_partition_space_used=18, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=[], build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14836, cpu_share=0.0, mem_res=12268, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=13840, cpu_share=0.0, mem_res=11464, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=31740, cpu_share=0.0, mem_res=30320, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=50956, cpu_share=0.0, mem_res=39452, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=6306056, free=10506184, buffers=688572, cached=15164300, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.32, five_min_avg=2.41, fifteen_min_avg=1.44, cpu_share=0, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/09/2025 02:15:01.193 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:01.193 7f520273b8d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 deleted = False process_info = [ << process_name = contrail-control process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1749435291000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-dns process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1749435291000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-named process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1749435292000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, << process_name = contrail-control-nodemgr process_state = PROCESS_STATE_RUNNING start_count = 1 stop_count = 0 exit_count = 0 last_start_time = 1749435291000000 last_stop_time = last_exit_time = core_file_list = [ ] >>, ] build_info = {"build-info" : [{"build-version" : "nightly", "build-time" : "2025-06-09 00:44:55.879161", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-9411-1.novalocal", "build-id" : "nightly", "build-number" : "nightly.el7"}]} >> 06/09/2025 02:15:01.193 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:01.193 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:01.194 7f520273b8d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 disk_usage_info = { /dev/vda1 : << partition_type = ext4 partition_space_used_1k = 14874584 partition_space_available_1k = 66215900 percentage_partition_space_used = 18 >> } all_core_file_list = [ ] process_mem_cpu_usage = { contrail-control : << mem_virt = 14836 cpu_share = 0.0 mem_res = 12268 >>contrail-dns : << mem_virt = 13840 cpu_share = 0.0 mem_res = 11464 >>contrail-named : << mem_virt = 31740 cpu_share = 0.0 mem_res = 30320 >>contrail-control-nodemgr : << mem_virt = 50956 cpu_share = 0.0 mem_res = 39452 >> } system_mem_usage = << total = 32665112 used = 6306056 free = 10506184 buffers = 688572 cached = 15164300 node_type = control-node >> system_cpu_usage = << one_min_avg = 3.32 five_min_avg = 2.41 fifteen_min_avg = 1.44 cpu_share = 0 node_type = control-node >> >> 06/09/2025 02:15:01.194 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:01.269 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:01.270 7f520273b8d0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:15:01.270 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:01.271 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:01.271 7f520273b8d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:01.271 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:01.271 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:15:01.271 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:15:01.271 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:01.271 7f520273b8d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:15:01.271 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:01.271 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:01.272 7f520273b8d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:15:01.272 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:05.272 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:05.273 7f520273b8d0 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:15:05.273 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:05.273 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:05.273 7f520273b8d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:05.274 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:05.274 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:15:05.274 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:15:05.274 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:05.274 7f520273b8d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:15:05.274 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:05.274 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:05.274 7f520273b8d0 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:15:05.274 7f520273b8d0 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:09.274 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:09.275 7f520225da20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:15:09.275 7f520225da20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:09.275 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:09.275 7f520225da20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:09.275 7f520225da20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:09.275 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:15:09.276 7f520225da20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:15:09.276 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:09.276 7f520225da20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:15:09.276 7f520225da20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:09.276 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:09.276 7f520225da20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:15:09.276 7f520225da20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:13.276 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:13.277 7f520225da20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:15:13.277 7f520225da20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:13.277 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:13.277 7f520225da20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:13.278 7f520225da20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:13.278 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:15:13.278 7f520225da20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:15:13.278 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:13.278 7f520225da20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:15:13.278 7f520225da20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:13.278 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:13.278 7f520225da20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:15:13.278 7f520225da20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:17.282 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:17.283 7f520225da20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:15:17.283 7f520225da20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:17.283 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:17.283 7f520225da20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:17.283 7f520225da20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:17.283 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:15:17.283 7f520225da20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:15:17.283 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:17.284 7f520225da20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:15:17.284 7f520225da20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:17.284 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:17.284 7f520225da20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:15:17.284 7f520225da20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:21.285 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:21.286 7f520225da20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:15:21.286 7f520225da20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:21.287 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:21.287 7f520225da20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:21.287 7f520225da20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:21.287 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:15:21.287 7f520225da20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:15:21.287 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:21.287 7f520225da20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:15:21.287 7f520225da20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:21.287 7f520225da20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:21.287 7f520225da20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:15:21.288 7f520225da20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:25.288 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:25.289 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:15:25.289 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:25.289 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:25.289 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:25.289 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:25.289 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:15:25.290 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:15:25.290 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:25.290 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:15:25.290 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:25.290 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:25.290 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:15:25.290 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:29.292 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:29.293 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:15:29.293 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:29.294 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:29.294 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:29.294 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:29.294 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:15:29.294 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:15:29.294 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:29.294 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:15:29.294 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:29.295 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:29.295 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:15:29.295 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:33.297 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:33.299 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:15:33.299 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:33.299 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:33.299 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:33.299 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:33.299 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:15:33.299 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:15:33.300 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:33.300 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:15:33.300 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:33.300 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:33.300 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:15:33.300 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:37.301 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:37.302 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:15:37.302 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:37.302 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:37.303 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:37.303 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:37.303 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:15:37.303 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:15:37.303 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:37.303 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:15:37.303 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:37.303 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:37.303 7f51fe104a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:15:37.303 7f51fe104a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:41.304 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:41.306 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:15:41.306 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:41.306 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:41.306 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:41.306 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:41.306 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:15:41.306 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:15:41.306 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:41.307 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:15:41.307 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:41.307 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:41.307 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:15:41.307 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:45.308 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:45.309 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:15:45.309 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:45.309 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:45.309 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:45.309 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:45.309 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:15:45.310 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:15:45.310 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:45.310 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:15:45.310 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:45.310 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:45.310 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:15:45.310 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:49.312 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:49.313 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:15:49.313 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:49.313 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:49.313 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:49.314 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:49.314 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:15:49.314 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:15:49.314 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:49.314 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:15:49.314 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:49.314 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:49.314 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:15:49.314 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:53.317 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:53.318 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: Session Event: TCP Connect Fail 06/09/2025 02:15:53.318 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:53.318 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:53.318 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:53.318 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:53.318 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 06/09/2025 02:15:53.319 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 06/09/2025 02:15:53.319 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:53.319 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 06/09/2025 02:15:53.319 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:53.319 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:53.319 7f51fe10ca20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1:Control:contrail-control-nodemgr:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8101 start_time = 1749435293267228 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 06/09/2025 02:15:53.319 7f51fe10ca20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 06/09/2025 02:15:57.319 7f51fd449a20 [contrail-control-nodemgr] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 06/09/2025 02:15:57.322 7f51fd449a20 [contrail-control-nodemgr] [INFO]: Session Event: TCP Connected 06/09/2025 02:15:57.323 7f51fd449a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 06/09/2025 02:15:57.323 7f51fd449a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:57.323 7f51fd449a20 [contrail-control-nodemgr] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3422-1 process_status = [ << module_id = contrail-control-nodemgr instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 06/09/2025 02:15:57.323 7f51fd449a20 [contrail-control-nodemgr] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 06/09/2025 02:15:57.323 7f51fd449a20 [contrail-control-nodemgr] [INFO]: Processing event[EvTcpConnected] in state[Connect] 06/09/2025 02:15:57.323 7f51fd449a20 [contrail-control-nodemgr] [DEBUG]: Send sandesh control message. uve type count # 6 06/09/2025 02:15:57.323 7f51fd449a20 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 06/09/2025 02:15:57.323 7f51fd449a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/09/2025 02:15:57.324 7f51fd449a20 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/09/2025 02:16:01.196 7f5202860388 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3422-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=15802808, partition_space_available_1k=65287676, percentage_partition_space_used=19, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15776, cpu_share=0.01, mem_res=12972, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14816, cpu_share=0.01, mem_res=12300, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=31740, cpu_share=0.0, mem_res=30320, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=51012, cpu_share=0.01, mem_res=39452, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=7906600, free=8074100, buffers=707520, cached=15976892, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=2.74, five_min_avg=2.41, fifteen_min_avg=1.5, cpu_share=3.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/09/2025 02:16:01.196 7f520223fac8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 06/09/2025 02:16:05.784 7f5202245eb8 [contrail-control-nodemgr] [DEBUG]: Extracted sandesh length: 0000001016 06/09/2025 02:16:05.785 7f5202245eb8 [contrail-control-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 06/09/2025 02:16:05.785 7f520007aeb8 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 06/09/2025 02:16:05.785 7f520007aeb8 [contrail-control-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 0 06/09/2025 02:16:05.785 7f520007aeb8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/09/2025 02:16:05.786 7f520007aeb8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/09/2025 02:16:05.786 7f520007aeb8 [contrail-control-nodemgr] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 06/09/2025 02:16:05.786 7f520223fac8 [contrail-control-nodemgr] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 06/09/2025 02:16:05.788 7f520223fac8 [contrail-control-nodemgr] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 06/09/2025 02:16:15.798 7f5202245eb8 [contrail-control-nodemgr] [DEBUG]: Extracted sandesh length: 0000001296 06/09/2025 02:16:15.798 7f5202245eb8 [contrail-control-nodemgr] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 06/09/2025 02:16:15.799 7f51fe10ccc0 [contrail-control-nodemgr] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 06/09/2025 02:16:15.799 7f51fe10ccc0 [contrail-control-nodemgr] [DEBUG]: Number of uve types in sandesh control message is 2 06/09/2025 02:16:15.799 7f51fe10ccc0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/09/2025 02:16:15.799 7f51fe10ccc0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 06/09/2025 02:17:01.211 7f5202860388 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3422-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=17523800, partition_space_available_1k=63566684, percentage_partition_space_used=22, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15664, cpu_share=0.01, mem_res=13208, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=14812, cpu_share=0.01, mem_res=12648, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=31568, cpu_share=0.0, mem_res=30320, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=41988, cpu_share=0.01, mem_res=39700, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=9129772, free=20219684, buffers=747936, cached=2567720, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.85, five_min_avg=2.84, fifteen_min_avg=1.71, cpu_share=4.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/09/2025 02:18:00.007 7f51fe1088d0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/09/2025 02:18:00.841 7f5202860388 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3422-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=17557172, partition_space_available_1k=63533312, percentage_partition_space_used=22, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16528, cpu_share=0.01, mem_res=13960, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15176, cpu_share=0.01, mem_res=12912, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=32056, cpu_share=0.0, mem_res=30792, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42056, cpu_share=0.01, mem_res=39736, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=9594652, free=19315100, buffers=757044, cached=2998316, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.2, five_min_avg=2.85, fifteen_min_avg=1.79, cpu_share=3.88, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/09/2025 02:18:02.110 7f51fe118a20 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/09/2025 02:18:51.818 7f51fd4546d8 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/09/2025 02:19:00.850 7f5202860388 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3422-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19062220, partition_space_available_1k=62028264, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16964, cpu_share=0.01, mem_res=14380, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15308, cpu_share=0.01, mem_res=13028, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=32056, cpu_share=0.0, mem_res=30792, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42112, cpu_share=0.01, mem_res=39788, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=9840632, free=17752964, buffers=787440, cached=4284076, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.52, five_min_avg=2.95, fifteen_min_avg=1.89, cpu_share=4.0, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/09/2025 02:20:00.870 7f5202860388 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3422-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19111860, partition_space_available_1k=61978624, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18864, cpu_share=0.02, mem_res=15696, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=15752, cpu_share=0.01, mem_res=13364, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=32076, cpu_share=0.0, mem_res=30812, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42132, cpu_share=0.01, mem_res=39804, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=10423860, free=17087076, buffers=787940, cached=4366236, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=7.59, five_min_avg=4.09, fifteen_min_avg=2.34, cpu_share=5.12, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/09/2025 02:21:00.877 7f5202860388 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3422-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19149880, partition_space_available_1k=61940604, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=21684, cpu_share=0.03, mem_res=17428, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16456, cpu_share=0.02, mem_res=13868, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=32084, cpu_share=0.0, mem_res=30816, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42152, cpu_share=0.01, mem_res=39816, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=10833840, free=16649360, buffers=787972, cached=4393940, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.51, five_min_avg=4.1, fifteen_min_avg=2.45, cpu_share=4.5, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/09/2025 02:22:00.886 7f5202860388 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3422-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19237376, partition_space_available_1k=61853108, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=22844, cpu_share=0.02, mem_res=18140, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=16816, cpu_share=0.01, mem_res=14132, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=32088, cpu_share=0.0, mem_res=30820, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42164, cpu_share=0.01, mem_res=39832, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=11429124, free=15991904, buffers=788076, cached=4456008, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=5.0, five_min_avg=4.32, fifteen_min_avg=2.64, cpu_share=4.75, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/09/2025 02:23:00.909 7f5202860388 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3422-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19235360, partition_space_available_1k=61855124, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=24620, cpu_share=0.02, mem_res=19016, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=17904, cpu_share=0.02, mem_res=14288, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=50404, cpu_share=0.0, mem_res=49064, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42172, cpu_share=0.01, mem_res=39832, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=10978784, free=16419980, buffers=788128, cached=4478220, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=4.91, five_min_avg=4.53, fifteen_min_avg=2.82, cpu_share=4.0, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/09/2025 02:24:00.909 7f5202860388 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3422-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19259244, partition_space_available_1k=61831240, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=25664, cpu_share=0.01, mem_res=19672, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18132, cpu_share=0.01, mem_res=14436, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=50144, cpu_share=0.0, mem_res=48792, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42172, cpu_share=0.01, mem_res=39832, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=10502160, free=16867160, buffers=788340, cached=4507452, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.57, five_min_avg=4.24, fifteen_min_avg=2.83, cpu_share=4.38, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/09/2025 02:25:00.920 7f5202860388 [contrail-control-nodemgr] [DEBUG]: DBG: event_tick_60: node_status=NodeStatus(name='cn-jenkins-deploy-platform-ansible-os-3422-1', deleted=None, status=None, process_status=None, process_info=None, disk_usage_info={'/dev/vda1': DiskPartitionUsageStats(partition_type='ext4', partition_name=None, partition_space_used_1k=19364148, partition_space_available_1k=61726336, percentage_partition_space_used=24, _LinuxSysData__key='/dev/vda1')}, description=None, all_core_file_list=None, build_info=None, process_mem_cpu_usage={'contrail-control': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=26572, cpu_share=0.01, mem_res=20108, _EventManager__key='contrail-control'), 'contrail-dns': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=18384, cpu_share=0.01, mem_res=14604, _EventManager__key='contrail-dns'), 'contrail-named': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=50144, cpu_share=0.0, mem_res=48792, _EventManager__key='contrail-named'), 'contrail-control-nodemgr': ProcessCpuInfo(module_id=None, inst_id=None, mem_virt=42168, cpu_share=0.01, mem_res=39832, _EventManager__key='contrail-control-nodemgr')}, system_cpu_info=None, system_mem_usage=SysMemInfo(total=32665112, used=10444352, free=16805656, buffers=788448, cached=4626656, node_type='control-node'), system_cpu_usage=SysCpuInfo(one_min_avg=3.68, five_min_avg=4.15, fifteen_min_avg=2.89, cpu_share=5.0, node_type='control-node'), running_package_version=None, installed_package_version=None, _table='ObjectBgpRouter') 06/09/2025 02:25:01.580 7f520273b8d0 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map 06/09/2025 02:25:42.784 7f51fd432d68 [contrail-control-nodemgr] [DEBUG]: send sync_uve in the [ObjectBgpRouter:NodeStatusUVE] map